diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index b2e7400d..550acca4 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1,29 +1,32 @@ # OpenSearch SDK for Java Developer Guide -[OpenSearch SDK for Java Developer Guide](#opensearch-sdk-for-java-developer-guide) -* [Getting started](#getting-started) - * [Start the extension](#start-the-extension) - * [Clone the OpenSearch SDK for Java repository](#clone-the-opensearch-sdk-for-java-repository) - * [Run the sample extension](#run-the-sample-extension) - * [Run the sample extension with security](#run-the-sample-extension-with-security) - * [Start OpenSearch](#start-opensearch) - * [Clone the OpenSearch repository](#clone-the-opensearch-repository) - * [Enable the extensions feature flag](#enable-the-extensions-feature-flag) - * [Option 1](#option-1) - * [Option 2](#option-2) - * [Option 3](#option-3) - * [Run OpenSearch](#run-opensearch) - * [Send a REST request to the extension](#send-a-rest-request-to-the-extension) -* [Developing your own extension](#developing-your-own-extension) - * [Running a custom extension](#running-a-custom-extension) - * [Publishing the OpenSearch SDK for Java repo to the Maven local repo](#publishing-the-opensearch-sdk-for-java-repo-to-the-maven-local-repo) - * [Running tests](#running-tests) - * [Launching and debugging from an IDE](#launching-and-debugging-from-an-ide) - * [Generating an artifact](#generating-an-artifact) - * [Submitting changes](#submitting-changes) - -## Getting started +- [OpenSearch SDK for Java Developer Guide](#opensearch-sdk-for-java-developer-guide) + - [Getting Started](#getting-started) + - [Start the Extension](#start-the-extension) + - [Clone the OpenSearch SDK for Java Repository](#clone-the-opensearch-sdk-for-java-repository) + - [Run the Sample Extension](#run-the-sample-extension) + - [Run the Sample Extension with Security](#run-the-sample-extension-with-security) + - [Start OpenSearch](#start-opensearch) + - [Clone the OpenSearch Repository](#clone-the-opensearch-repository) + - [Enable the Extensions Feature Flag](#enable-the-extensions-feature-flag) + - [Option 1](#option-1) + - [Option 2](#option-2) + - [Option 3](#option-3) + - [Run OpenSearch](#run-opensearch) + - [Using Gradle](#using-gradle) + - [From a Binary Distribution](#from-a-binary-distribution) + - [Extension Registration](#extension-registration) + - [Send a REST Request to the Extension](#send-a-rest-request-to-the-extension) + - [Developing Your Own Extension](#developing-your-own-extension) + - [Running a Custom Extension](#running-a-custom-extension) + - [Publishing the OpenSearch SDK for Java Repo to Maven Local](#publishing-the-opensearch-sdk-for-java-repo-to-maven-local) + - [Running Tests](#running-tests) + - [Launching and Debugging from an IDE](#launching-and-debugging-from-an-ide) + - [Generating an Artifact](#generating-an-artifact) + - [Contributing Changes](#contributing-changes) + +## Getting Started In general, running and using an extension can be broken down into the following steps: @@ -42,11 +45,11 @@ Note: You need to first start the extension or extensions and then start OpenSea This tutorial uses the sample Hello World extension included in the `opensearch-sdk-java` repository. -### Start the extension +### Start the Extension To start the extension, you need to first clone the OpenSearch SDK for Java repository and then run the extension. -#### Clone the OpenSearch SDK for Java repository +#### Clone the OpenSearch SDK for Java Repository Fork the [OpenSearch SDK for Java](https://github.com/opensearch-project/opensearch-sdk-java) repository and clone it locally using the following command: @@ -54,7 +57,7 @@ Fork the [OpenSearch SDK for Java](https://github.com/opensearch-project/opensea git clone https://github.com//opensearch-sdk-java.git ``` -#### Run the sample extension +#### Run the Sample Extension Navigate to the directory to which you cloned the OpenSearch SDK for Java repository. @@ -71,7 +74,9 @@ Bound addresses will then be logged to the terminal: [main] INFO transportservice.TransportService - profile [test]: publish_address {127.0.0.1:5555}, bound_addresses {[::1]:5555}, {127.0.0.1:5555} ``` -#### Run the sample extension with security +#### Run the Sample Extension with Security + +This is not required. 1. Uncomment the SSL settings from [resources/sample/helloworld-settings.yml](src/main/resources/sample/helloworld-settings.yml): ``` @@ -82,8 +87,8 @@ ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem ssl.transport.enforce_hostname_verification: false path.home: ``` -2. Follow the instructions in [CERTIFICATE_GENERATION](Docs/CERTIFICATE_GENERATION.md) to generate the certificates. -3. Run the extension using `./gradlew run`. +1. Follow the instructions in [CERTIFICATE_GENERATION](Docs/CERTIFICATE_GENERATION.md) to generate the certificates. +2. Run the extension using `./gradlew run`. ### Start OpenSearch @@ -92,7 +97,7 @@ Follow these steps to start OpenSearch: - [Enable the extensions feature flag](#enable-the-extensions-feature-flag). - [Run OpenSearch](#run-opensearch). -#### Clone the OpenSearch repository +#### Clone the OpenSearch Repository Fork the [OpenSearch](https://github.com/opensearch-project/OpenSearch/) repository and clone it locally using the following command: @@ -100,9 +105,10 @@ Fork the [OpenSearch](https://github.com/opensearch-project/OpenSearch/) reposit git clone https://github.com//OpenSearch.git ``` -#### Enable the extensions feature flag +#### Enable the Extensions Feature Flag -Extensions are experimental in OpenSearch 2.9, so you must enable them either before or when you run OpenSearch. You can enable the feature flag using one of the following options. +Extensions are experimental since OpenSearch 2.9, so you must enable them either before or when you run OpenSearch. You can enable the feature flag using one of the +following options. ##### Option 1 @@ -136,44 +142,30 @@ Enable the experimental feature flag by setting it to `true` in `opensearch.yml` #### Run OpenSearch -You can run OpenSearch either from a compiled binary or from Gradle. +You can run OpenSearch either from a compiled binary or from source code. -To **run OpenSearch from a compiled binary**, follow these steps: +##### Using Gradle -- Start a separate terminal and navigate to the directory where OpenSearch has been cloned using `cd OpenSearch`. -- Run `./gradlew assemble` to create a local distribution. -- Start OpenSearch using `./bin/opensearch`. -- Send the below sample REST API to initialize an extension -```bash -curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data '{ \ -"name":"hello-world", \ -"uniqueId":"hello-world", \ -"hostAddress":"127.0.0.1", \ -"port":"4532", \ -"version":"1.0", \ -"opensearchVersion":"3.0.0", \ -"minimumCompatibleVersion":"3.0.0", \ -"dependencies":[{"uniqueId":"test1","version":"2.0.0"},{"uniqueId":"test2","version":"3.0.0"}] \ -}' -``` +If you closed OpenSearch as above, use `gradle` to run it. -To **run OpenSearch from Gradle**, follow these steps: - Run `./gradlew run` to start OpenSearch. -- Send the below sample REST API to initialize an extension +- Send the below sample REST API to initialize an extension using settings in [hello.json](src/main/java/org/opensearch/sdk/sample/helloworld/hello.json). + ```bash -curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data '{ \ -"name":"hello-world", \ -"uniqueId":"hello-world", \ -"hostAddress":"127.0.0.1", \ -"port":"4532", \ -"version":"1.0", \ -"opensearchVersion":"3.0.0", \ -"minimumCompatibleVersion":"3.0.0", \ -"dependencies":[{"uniqueId":"test1","version":"2.0.0"},{"uniqueId":"test2","version":"3.0.0"}] \ -}' +curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data @src/main/java/org/opensearch/sdk/sample/helloworld/hello.json ``` -In response to the REST `/initialize` request, `ExtensionsManager` discovers the extension listening on a predefined port and executes the TCP handshake protocol to establish a data transfer connection. Then OpenSearch sends a request to the OpenSearch SDK for Java and, upon acknowledgment, the extension responds with its name. This name is logged in the terminal where OpenSearch is running: +##### From a Binary Distribution + +You can also build OpenSearch with `./gradlew assemble` or download a released version and start it using `./bin/opensearch`. + +Note: If the Security plugin is initialized in OpenSearch, use admin credentials to send extension initialization request. + +#### Extension Registration + +In response to the REST `/initialize` request, `ExtensionsManager` discovers the extension listening on a predefined port and executes the TCP handshake protocol to +establish a data transfer connection. Then OpenSearch sends a request to the OpenSearch SDK for Java and, upon acknowledgment, the extension responds with its name. This +name is logged in the terminal where OpenSearch is running: ```bash [2022-06-16T21:30:18,857][INFO ][o.o.t.TransportService ] [runTask-0] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} @@ -188,7 +180,8 @@ The OpenSearch SDK terminal also logs all requests and responses it receives fro - TCP handshake request: ```bash -21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600, L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B +21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600, +L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ @@ -203,7 +196,8 @@ MESSAGE RECEIVED:E«󀀀internal:tcp/handshake£·A - Extension name request/response: ```bash -21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B +21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, +L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ @@ -221,13 +215,17 @@ MESSAGE RECEIVED:E«󀀀internal:tcp/handshake£·A |000000b0| 65 6d 6f 74 65 5f 63 6c 75 73 74 65 72 5f 63 6c |emote_cluster_cl| |000000c0| 69 65 6e 74 01 72 00 a3 8e b7 41 00 |ient.r....A. | +--------+-------------------------------------------------+----------------+ -MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX4bkVjG-Q 127.0.0.1 127.0.0.1 127.0.0.1´cluster_managermdatadingestiremote_cluster_clientr£·A -21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 204, request id: 3, type: request, version: 3.0.0, action: internal:discovery/extensions] READ: 204B +MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX4bkVjG-Q 127.0.0.1 127.0.0.1 +127.0.0.1´cluster_managermdatadingestiremote_cluster_clientr£·A +21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, +remoteAddress=/127.0.0.1:47782} [length: 204, request id: 3, type: request, version: 3.0.0, action: internal:discovery/extensions] READ: 204B 21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] received request 21:30:18.996 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - register 2 [transport] [internal:discovery/extensions] [] 21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - unregister task for id: 2 -21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 48, request id: 3, type: response, version: 3.0.0, header size: 2B] WRITE: 48B -21:30:18.998 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] WRITE: 48B +21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, +remoteAddress=/127.0.0.1:47782} [length: 48, request id: 3, type: response, version: 3.0.0, header size: 2B] WRITE: 48B +21:30:18.998 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, +L:/127.0.0.1:4532 - R:/127.0.0.1:47782] WRITE: 48B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ @@ -235,30 +233,32 @@ MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX |00000010| 2d c7 23 00 00 00 02 00 00 16 4e 61 6d 65 64 57 |-.#.......exampl| |00000020| 72 69 74 65 61 62 6c 65 52 65 67 69 73 74 72 79 |epluginname | +--------+-------------------------------------------------+----------------+ -21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] FLUSH +21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, +L:/127.0.0.1:4532 - R:/127.0.0.1:47782] FLUSH 21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] sent response ``` -It is important to ensure that the OpenSearch SDK for Java is already running on a separate process. +### Send a REST Request to the Extension -### Send a REST request to the extension +The following request is configured to be handled by the sample `HelloWorldExtension` (note that its matching `uniqueId` is `hello-world-java`): -The following request is configured to be handled by the sample `HelloWorldExtension` (note that its matching `uniqueId` is `opensearch-sdk-java-1`): ```bash -curl -X GET localhost:9200/_extensions/_opensearch-sdk-java-1/hello +curl -X GET localhost:9200/_extensions/_hello-world-java/hello ``` -## Developing your own extension +## Developing Your Own Extension -Before you write your own extension, read through the [design documentation](DESIGN.md) to learn about extension architecture and class hierarchy. Then follow [this guide](CREATE_YOUR_FIRST_EXTENSION.md) to develop your own extension. For an example, see the sample Hello World extension in the `org.opensearch.sdk.sample.helloworld` package. +Before you write your own extension, read through the [design documentation](DESIGN.md) to learn about extension architecture and class hierarchy. Then follow [this +guide](CREATE_YOUR_FIRST_EXTENSION.md) to develop your own extension. For an example, see the sample Hello World extension in the `org.opensearch.sdk.sample.helloworld` +package. Refer to the following sections for information about post-development tasks. -### Running a custom extension +### Running a Custom Extension To run an extension that uses the SDK, use `./gradlew run` on that extension. -### Publishing the OpenSearch SDK for Java repo to the Maven local repo +### Publishing the OpenSearch SDK for Java Repo to Maven Local Until we publish this repo to the Maven Central Repository, publishing to the Maven local repository is how extensions (outside of sample packages) import the artifacts: @@ -266,21 +266,23 @@ Until we publish this repo to the Maven Central Repository, publishing to the Ma ./gradlew publishToMavenLocal ``` -### Running tests +### Running Tests Use the following command to run tests: -``` +```bash ./gradlew clean build integTest ``` -### Launching and debugging from an IDE +### Launching and Debugging from an IDE -For information about launching and debugging from an IDE in OpenSearch, see [this document](https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#launching-and-debugging-from-an-ide) +For information about launching and debugging from an IDE in OpenSearch, see [this +document](https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#launching-and-debugging-from-an-ide) -### Generating an artifact +### Generating an Artifact -In `opensearch-sdk-java`, navigate to `build/distributions`. Look for the tarball in the form `opensearch-sdk-java-1.0.0-SNAPSHOT.tar`. If there is no such tarball, use the following command to create one: +In `opensearch-sdk-java`, navigate to `build/distributions`. Look for the tarball in the form `opensearch-sdk-java-1.0.0-SNAPSHOT.tar`. If there is no such tarball, use +the following command to create one: ```bash ./gradlew clean && ./gradlew build ``` @@ -300,8 +302,11 @@ The artifact will include extension settings for the sample Hello World extensio opensearchPort: 9200 ``` +You can optionally add `routeNamePrefix:` as a value to the yml. This setting allows you to prefix all your registered NamedRoute names. +The value must be alphanumeric and can contain `_` in the name. + Start the sample extension with `./bin/opensearch-sdk-java` -### Submitting changes +### Contributing Changes To learn how to submit your changes, see [CONTRIBUTING](CONTRIBUTING.md). diff --git a/src/main/java/org/opensearch/sdk/sample/helloworld/hello.json b/src/main/java/org/opensearch/sdk/sample/helloworld/hello.json new file mode 100644 index 00000000..daf8c599 --- /dev/null +++ b/src/main/java/org/opensearch/sdk/sample/helloworld/hello.json @@ -0,0 +1,19 @@ +{ + "name": "Hello World", + "uniqueId": "hello-world-java", + "hostAddress": "127.0.0.1", + "port": "4500", + "version": "1.0", + "opensearchVersion": "3.0.0", + "minimumCompatibleVersion": "3.0.0", + "dependencies": [ + { + "uniqueId": "test1", + "version": "2.0.0" + }, + { + "uniqueId": "test2", + "version": "3.0.0" + } + ] +}