Merge branch 'main' into opensearch-api-async

Signed-off-by: Souvik Bose <[email protected]>

CODE_OF_CONDUCT.md

@@ -1,5 +1,5 @@
-This code of conduct applies to all spaces provided by the OpenSource project including in code, documentation, issue trackers, mailing lists, chat channels, wikis, blogs, social media and any other communication channels used by the project.
 
+This code of conduct applies to all spaces provided by the OpenSource project including in code, documentation, issue trackers, mailing lists, chat channels, wikis, blogs, social media, events, conferences, meetings, and any other communication channels used by the project.
 
 **Our open source communities endeavor to:**
 
@@ -8,7 +8,6 @@ This code of conduct applies to all spaces provided by the OpenSource project in
 * Be Respectful: We are committed to encouraging differing viewpoints, accepting constructive criticism and work collaboratively towards decisions that help the project grow. Disrespectful and unacceptable behavior will not be tolerated.
 * Be Collaborative: We are committed to supporting what is best for our community and users. When we build anything for the benefit of the project, we should document the work we do and communicate to others on how this affects their work.
 
-
 **Our Responsibility. As contributors, members, or bystanders we each individually have the responsibility to behave professionally and respectfully at all times. Disrespectful and unacceptable behaviors include, but are not limited to:**
 
 * The use of violent threats, abusive, discriminatory, or derogatory language;
@@ -19,6 +18,7 @@ This code of conduct applies to all spaces provided by the OpenSource project in
 * Publishing private information, such as physical or electronic address, without permission;
 * Other conduct which could reasonably be considered inappropriate in a professional setting;
 * Advocating for or encouraging any of the above behaviors.
-* Enforcement and Reporting Code of Conduct Issues:
+
+**Enforcement and Reporting Code of Conduct Issues:**
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be reported. [Contact us](mailto:[email protected]). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances.

TRIAGING.md

@@ -0,0 +1,73 @@
+<img src="https://raw.githubusercontent.com/opensearch-project/data-prepper/main/docs/images/DataPrepper_auto.svg" height="64px" alt="Data Prepper">
+
+The Data Prepper maintainers seek to promote an inclusive and engaged community of contributors.
+In order to facilitate this, weekly triage meetings are open to all and attendance is encouraged for anyone who hopes to contribute, discuss an issue, or learn more about the project.
+To learn more about contributing to the Data Prepper project visit the [Contributing](./CONTRIBUTING.md) documentation.
+
+### Do I need to attend for my issue to be addressed/triaged?
+
+Attendance is not required for your issue to be triaged or addressed.
+All new issues are triaged weekly.
+
+### What happens if my issue does not get covered this time?
+
+Each meeting we seek to address all new issues.
+However, should we run out of time before your issue is discussed, you are always welcome to attend the next meeting or to follow up on the issue post itself.
+
+### How do I join the triage meeting?
+
+Meetings are hosted regularly Tuesdays at 2:30 PM US Central Time (12:30 PM Pacific Time) and can be joined via the links posted on the [OpenSearch Meetup Group](https://www.meetup.com/opensearch/events/) list of events.
+The event will be titled `Data Prepper Triage Meeting`.
+
+After joining the Zoom meeting, you can enable your video / voice to join the discussion.
+If you do not have a webcam or microphone available, you can still join in via the text chat.
+
+If you have an issue you'd like to bring forth please consider getting a link to the issue so it can be presented to everyone in the meeting.
+
+### Is there an agenda for each week?
+
+Meetings are 30 minutes and structured as follows:
+
+1. Initial Gathering: As we gather, feel free to turn on video and engage in informal and open-to-all conversation. A volunteer Data Prepper maintainer will share the [Data Prepper Tracking Board](https://github.com/orgs/opensearch-project/projects/82/) and proceed.
+2. Announcements: We will make any announcements at the beginning, if necessary.
+3. Untriaged issues: We will review all untriaged [issues](https://github.com/orgs/opensearch-project/projects/82/views/6) for the Data Prepper repository. If you have an item here, you may spend a few minutes to explain your request.
+4. Member Requests: Opportunity for any meeting member to ask for consideration of an issue or pull request.
+5. Release review: If time permits, and we find it necessary, we will review [items for the current release](https://github.com/orgs/opensearch-project/projects/82/views/14).
+6. Follow-up issues: If time permits, we will review the [follow up items](https://github.com/orgs/opensearch-project/projects/82/views/18).
+7. Open Discussion: If time permits, allow for members of the meeting to surface any topics without issues filed or pull request created.
+
+### Do I need to have already contributed to the project to attend a triage meeting?
+
+No, all are welcome and encouraged to attend.
+Attending the triage meetings is a great way for a new contributor to learn about the project as well as explore different avenues of contribution.
+
+### What if I have follow-up questions on an issue?
+
+If you have an existing issue you would like to discuss, you can always comment on the issue itself.
+Alternatively, you are welcome to come to the triage meeting to discuss.
+
+### Is this meeting a good place to get help using Data Prepper?
+
+While we are always happy to help the community, the best resource for usage questions is the [the Data Prepper discussion forum](https://github.com/opensearch-project/data-prepper/discussions) on GitHub.
+
+There you can find answers to many common questions as well as speak with implementation experts and Data Prepper maintainers.
+
+### What are the issue labels associated with triaging?
+
+There are several labels that are particularly important for triaging in Data Prepper:
+
+| Label | When applied | Meaning |
+| ----- | ------------ | ------- |
+| [untriaged](https://github.com/opensearch-project/data-prepper/labels/untriaged) | When issues are created or re-opened. | Issues labeled as `untriaged` require the attention of the repository maintainers and may need to be prioritized for quicker resolution. It's crucial to keep the count of 'untriaged' labels low to ensure all potential security issues are addressed in a timely manner. |
+| [follow up](https://github.com/opensearch-project/data-prepper/labels/follow%20up) | During triage meetings. | Issues labeled as `follow up` have been triaged. However, the maintainers may need to follow up further on it. This tag lets us triage an issue as not critical, but also be able to come back to it.
+| [help wanted](https://github.com/opensearch-project/data-prepper/labels/help%20wanted) | Anytime. | Issues marked as `help wanted` signal that they are actionable and not the current focus of the project maintainers. Community contributions are especially encouraged for these issues. |
+| [good first issue](https://github.com/opensearch-project/data-prepper/labels/good%20first%20issue) | Anytime. | Issues labeled as `good first issue` are small in scope and can be resolved with a single pull request. These are recommended starting points for newcomers looking to make their first contributions. |
+
+
+### Is this where I should bring up potential security vulnerabilities?
+
+Due to the sensitive nature of security vulnerabilities, please report all potential vulnerabilities directly by following the steps outlined in the [Security Issue Response Process](https://github.com/opensearch-project/data-prepper/security/policy).
+
+### Who should I contact if I have further questions?
+
+You can always file an [issue](https://github.com/opensearch-project/data-prepper/issues/new/choose) for any question you have about the project.

build.gradle

@@ -69,7 +69,7 @@ subprojects {
         }
     }
     dependencies {
-        implementation platform('com.fasterxml.jackson:jackson-bom:2.16.1')
+        implementation platform('com.fasterxml.jackson:jackson-bom:2.17.2')
         implementation platform('org.eclipse.jetty:jetty-bom:9.4.53.v20231009')
         implementation platform('io.micrometer:micrometer-bom:1.10.5')
         implementation libs.guava.core
@@ -226,6 +226,9 @@ subprojects {
 
     test {
         useJUnitPlatform()
+        javaLauncher = javaToolchains.launcherFor {
+            languageVersion = JavaLanguageVersion.current()
+        }
         reports {
             junitXml.required
             html.required

data-prepper-api/build.gradle

@@ -12,11 +12,11 @@ dependencies {
     implementation 'com.fasterxml.jackson.core:jackson-databind'
     implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310'
     implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8'
-    implementation 'org.apache.parquet:parquet-common:1.14.0'
+    implementation libs.parquet.common
     testImplementation 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml'
     implementation libs.commons.lang3
     testImplementation project(':data-prepper-test-common')
-    testImplementation 'org.skyscreamer:jsonassert:1.5.1'
+    testImplementation 'org.skyscreamer:jsonassert:1.5.3'
     testImplementation libs.commons.io
 }
 

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/acknowledgements/AcknowledgementSetManager.java

@@ -5,9 +5,6 @@
 
 package org.opensearch.dataprepper.model.acknowledgements;
 
-import org.opensearch.dataprepper.model.event.Event;
-import org.opensearch.dataprepper.model.event.EventHandle;
-
 import java.time.Duration;
 import java.util.function.Consumer;
 
@@ -29,32 +26,4 @@ public interface AcknowledgementSetManager {
      * @since 2.2
      */
     AcknowledgementSet create(final Consumer<Boolean> callback, final Duration timeout);
-
-    /**
-     * Releases an event's reference
-     *
-     * @param eventHandle event handle
-     * @param success indicates negative or positive acknowledgement
-     *
-     * @since 2.2
-     */
-    void releaseEventReference(final EventHandle eventHandle, boolean success);
-
-    /**
-     * Acquires an event's reference
-     *
-     * @param eventHandle event handle
-     *
-     * @since 2.2
-     */
-    void acquireEventReference(final EventHandle eventHandle);
-
-    /**
-     * Acquires an event's reference
-     *
-     * @param event event
-     *
-     * @since 2.2
-     */
-    void acquireEventReference(final Event event);
 }

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/configuration/PluginSetting.java

@@ -5,10 +5,22 @@
 
 package org.opensearch.dataprepper.model.configuration;
 
+import org.opensearch.dataprepper.model.annotations.DataPrepperPlugin;
+
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 
+/**
+ * Deprecated class for getting plugin settings.
+ * <p>
+ * Only projects within data-prepper-core should use this. It is currently used
+ * extensively in plugin framework to load plugins. In Data Prepper 3.0 this
+ * class will be moved into data-prepper-core and not exposed to plugins anymore.
+ *
+ * @deprecated Use {@link DataPrepperPlugin#pluginConfigurationType()} or {@link PipelineDescription} instead.
+ */
+@Deprecated
 public class PluginSetting implements PipelineDescription {
 
     private static final String UNEXPECTED_ATTRIBUTE_TYPE_MSG = "Unexpected type [%s] for attribute [%s]";

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/AbstractEventHandle.java

@@ -0,0 +1,52 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package org.opensearch.dataprepper.model.event;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.time.Instant;
+import java.util.function.BiConsumer;
+
+abstract class AbstractEventHandle implements EventHandle, InternalEventHandle {
+    private Instant externalOriginationTime;
+    private final Instant internalOriginationTime;
+    private List<BiConsumer<EventHandle, Boolean>> releaseConsumers;
+
+    AbstractEventHandle(final Instant internalOriginationTime) {
+        this.externalOriginationTime = null;
+        this.internalOriginationTime = internalOriginationTime;
+        this.releaseConsumers = new ArrayList<>();
+    }
+    @Override
+    public void setExternalOriginationTime(final Instant externalOriginationTime) {
+        this.externalOriginationTime = externalOriginationTime;
+    }
+
+    @Override
+    public Instant getInternalOriginationTime() {
+        return this.internalOriginationTime;
+    }
+
+    @Override
+    public Instant getExternalOriginationTime() {
+        return this.externalOriginationTime;
+    }
+
+    @Override
+    public void onRelease(BiConsumer<EventHandle, Boolean> releaseConsumer) {
+        synchronized (releaseConsumers) {
+            releaseConsumers.add(releaseConsumer);
+        }
+    }
+
+    public void notifyReleaseConsumers(boolean result) {
+        synchronized (releaseConsumers) {
+            for (final BiConsumer<EventHandle, Boolean> consumer: releaseConsumers) {
+                consumer.accept(this, result);
+            }
+        }
+    }
+}

data-prepper-api/src/main/java/org/opensearch/dataprepper/model/event/AggregateEventHandle.java

@@ -0,0 +1,77 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package org.opensearch.dataprepper.model.event;
+
+import org.opensearch.dataprepper.model.acknowledgements.AcknowledgementSet;
+import java.lang.ref.WeakReference;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.time.Instant;
+import java.io.Serializable;
+
+public class AggregateEventHandle extends AbstractEventHandle implements Serializable {
+    private List<WeakReference<AcknowledgementSet>> acknowledgementSetRefList;
+    private Set<Integer> acknowledgementSetHashes;
+
+    public AggregateEventHandle(final Instant internalOriginationTime) {
+        super(internalOriginationTime);
+        this.acknowledgementSetRefList = new ArrayList<>();
+        this.acknowledgementSetHashes = new HashSet<>();
+    }
+
+    @Override
+    public void addAcknowledgementSet(final AcknowledgementSet acknowledgementSet) {
+        int hashCode = acknowledgementSet.hashCode();
+        if (!acknowledgementSetHashes.contains(hashCode)) {
+            this.acknowledgementSetRefList.add(new WeakReference<>(acknowledgementSet));
+            acknowledgementSetHashes.add(hashCode);
+        }
+    }
+
+    @Override
+    public boolean hasAcknowledgementSet() {
+        return acknowledgementSetRefList.size() != 0;
+    }
+
+    @Override
+    public void acquireReference() {
+        synchronized (this) {
+            for (WeakReference<AcknowledgementSet> acknowledgementSetRef: acknowledgementSetRefList) {;
+                AcknowledgementSet acknowledgementSet = acknowledgementSetRef.get();
+                if (acknowledgementSet != null) {
+                    acknowledgementSet.acquire(this);
+                } 
+            }
+        }
+    }
+
+    @Override
+    public boolean release(boolean result) {
+        notifyReleaseConsumers(result);
+        boolean returnValue = true;
+        synchronized (this) {
+            for (WeakReference<AcknowledgementSet> acknowledgementSetRef: acknowledgementSetRefList) {
+                AcknowledgementSet acknowledgementSet = acknowledgementSetRef.get();
+                if (acknowledgementSet != null) {
+                    acknowledgementSet.release(this, result);
+                } else {
+                    returnValue = false;
+                }
+            }
+        }
+        return returnValue;
+    }
+
+    // For testing
+    List<WeakReference<AcknowledgementSet>> getAcknowledgementSetRefs() {
+        return acknowledgementSetRefList;
+    }
+
+}
+