Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 1.x] Updated dev guide with the correct GET route. #948

Merged
merged 2 commits into from
Oct 13, 2023
Merged

[Backport 1.x] Updated dev guide with the correct GET route. #948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
73840bf
Updated dev guide with the correct GET route. (#944)
dblock Oct 2, 2023
33f6bc0
chore(deps): update plugin com.diffplug.spotless to v6.22.0 (#946) (#…
opensearch-trigger-bot[bot] Oct 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 90 additions & 85 deletions DEVELOPER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,32 @@

# OpenSearch SDK for Java Developer Guide

[OpenSearch SDK for Java Developer Guide](#opensearch-sdk-for-java-developer-guide)
* [Getting started](#getting-started)
* [Start the extension](#start-the-extension)
* [Clone the OpenSearch SDK for Java repository](#clone-the-opensearch-sdk-for-java-repository)
* [Run the sample extension](#run-the-sample-extension)
* [Run the sample extension with security](#run-the-sample-extension-with-security)
* [Start OpenSearch](#start-opensearch)
* [Clone the OpenSearch repository](#clone-the-opensearch-repository)
* [Enable the extensions feature flag](#enable-the-extensions-feature-flag)
* [Option 1](#option-1)
* [Option 2](#option-2)
* [Option 3](#option-3)
* [Run OpenSearch](#run-opensearch)
* [Send a REST request to the extension](#send-a-rest-request-to-the-extension)
* [Developing your own extension](#developing-your-own-extension)
* [Running a custom extension](#running-a-custom-extension)
* [Publishing the OpenSearch SDK for Java repo to the Maven local repo](#publishing-the-opensearch-sdk-for-java-repo-to-the-maven-local-repo)
* [Running tests](#running-tests)
* [Launching and debugging from an IDE](#launching-and-debugging-from-an-ide)
* [Generating an artifact](#generating-an-artifact)
* [Submitting changes](#submitting-changes)

## Getting started
- [OpenSearch SDK for Java Developer Guide](#opensearch-sdk-for-java-developer-guide)
- [Getting Started](#getting-started)
- [Start the Extension](#start-the-extension)
- [Clone the OpenSearch SDK for Java Repository](#clone-the-opensearch-sdk-for-java-repository)
- [Run the Sample Extension](#run-the-sample-extension)
- [Run the Sample Extension with Security](#run-the-sample-extension-with-security)
- [Start OpenSearch](#start-opensearch)
- [Clone the OpenSearch Repository](#clone-the-opensearch-repository)
- [Enable the Extensions Feature Flag](#enable-the-extensions-feature-flag)
- [Option 1](#option-1)
- [Option 2](#option-2)
- [Option 3](#option-3)
- [Run OpenSearch](#run-opensearch)
- [Using Gradle](#using-gradle)
- [From a Binary Distribution](#from-a-binary-distribution)
- [Extension Registration](#extension-registration)
- [Send a REST Request to the Extension](#send-a-rest-request-to-the-extension)
- [Developing Your Own Extension](#developing-your-own-extension)
- [Running a Custom Extension](#running-a-custom-extension)
- [Publishing the OpenSearch SDK for Java Repo to Maven Local](#publishing-the-opensearch-sdk-for-java-repo-to-maven-local)
- [Running Tests](#running-tests)
- [Launching and Debugging from an IDE](#launching-and-debugging-from-an-ide)
- [Generating an Artifact](#generating-an-artifact)
- [Contributing Changes](#contributing-changes)

## Getting Started

In general, running and using an extension can be broken down into the following steps:

Expand All @@ -42,19 +45,19 @@ Note: You need to first start the extension or extensions and then start OpenSea

This tutorial uses the sample Hello World extension included in the `opensearch-sdk-java` repository.

### Start the extension
### Start the Extension

To start the extension, you need to first clone the OpenSearch SDK for Java repository and then run the extension.

#### Clone the OpenSearch SDK for Java repository
#### Clone the OpenSearch SDK for Java Repository

Fork the [OpenSearch SDK for Java](https://github.com/opensearch-project/opensearch-sdk-java) repository and clone it locally using the following command:

```bash
git clone https://github.com/<your username>/opensearch-sdk-java.git
```

#### Run the sample extension
#### Run the Sample Extension

Navigate to the directory to which you cloned the OpenSearch SDK for Java repository.

Expand All @@ -71,7 +74,9 @@ Bound addresses will then be logged to the terminal:
[main] INFO transportservice.TransportService - profile [test]: publish_address {127.0.0.1:5555}, bound_addresses {[::1]:5555}, {127.0.0.1:5555}
```

#### Run the sample extension with security
#### Run the Sample Extension with Security

This is not required.

1. Uncomment the SSL settings from [resources/sample/helloworld-settings.yml](src/main/resources/sample/helloworld-settings.yml):
```
Expand All @@ -82,8 +87,8 @@ ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
ssl.transport.enforce_hostname_verification: false
path.home: <path/to/extension>
```
2. Follow the instructions in [CERTIFICATE_GENERATION](Docs/CERTIFICATE_GENERATION.md) to generate the certificates.
3. Run the extension using `./gradlew run`.
1. Follow the instructions in [CERTIFICATE_GENERATION](Docs/CERTIFICATE_GENERATION.md) to generate the certificates.
2. Run the extension using `./gradlew run`.

### Start OpenSearch

Expand All @@ -92,17 +97,18 @@ Follow these steps to start OpenSearch:
- [Enable the extensions feature flag](#enable-the-extensions-feature-flag).
- [Run OpenSearch](#run-opensearch).

#### Clone the OpenSearch repository
#### Clone the OpenSearch Repository

Fork the [OpenSearch](https://github.com/opensearch-project/OpenSearch/) repository and clone it locally using the following command:

```bash
git clone https://github.com/<your username>/OpenSearch.git
```

#### Enable the extensions feature flag
#### Enable the Extensions Feature Flag

Extensions are experimental in OpenSearch 2.9, so you must enable them either before or when you run OpenSearch. You can enable the feature flag using one of the following options.
Extensions are experimental since OpenSearch 2.9, so you must enable them either before or when you run OpenSearch. You can enable the feature flag using one of the
following options.

##### Option 1

Expand Down Expand Up @@ -136,44 +142,30 @@ Enable the experimental feature flag by setting it to `true` in `opensearch.yml`
#### Run OpenSearch
You can run OpenSearch either from a compiled binary or from Gradle.
You can run OpenSearch either from a compiled binary or from source code.
To **run OpenSearch from a compiled binary**, follow these steps:
##### Using Gradle
- Start a separate terminal and navigate to the directory where OpenSearch has been cloned using `cd OpenSearch`.
- Run `./gradlew assemble` to create a local distribution.
- Start OpenSearch using `./bin/opensearch`.
- Send the below sample REST API to initialize an extension
```bash
curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data '{ \
"name":"hello-world", \
"uniqueId":"hello-world", \
"hostAddress":"127.0.0.1", \
"port":"4532", \
"version":"1.0", \
"opensearchVersion":"3.0.0", \
"minimumCompatibleVersion":"3.0.0", \
"dependencies":[{"uniqueId":"test1","version":"2.0.0"},{"uniqueId":"test2","version":"3.0.0"}] \
}'
```
If you closed OpenSearch as above, use `gradle` to run it.
To **run OpenSearch from Gradle**, follow these steps:
- Run `./gradlew run` to start OpenSearch.
- Send the below sample REST API to initialize an extension
- Send the below sample REST API to initialize an extension using settings in [hello.json](src/main/java/org/opensearch/sdk/sample/helloworld/hello.json).
```bash
curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data '{ \
"name":"hello-world", \
"uniqueId":"hello-world", \
"hostAddress":"127.0.0.1", \
"port":"4532", \
"version":"1.0", \
"opensearchVersion":"3.0.0", \
"minimumCompatibleVersion":"3.0.0", \
"dependencies":[{"uniqueId":"test1","version":"2.0.0"},{"uniqueId":"test2","version":"3.0.0"}] \
}'
curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data @src/main/java/org/opensearch/sdk/sample/helloworld/hello.json
```
In response to the REST `/initialize` request, `ExtensionsManager` discovers the extension listening on a predefined port and executes the TCP handshake protocol to establish a data transfer connection. Then OpenSearch sends a request to the OpenSearch SDK for Java and, upon acknowledgment, the extension responds with its name. This name is logged in the terminal where OpenSearch is running:
##### From a Binary Distribution
You can also build OpenSearch with `./gradlew assemble` or download a released version and start it using `./bin/opensearch`.
Note: If the Security plugin is initialized in OpenSearch, use admin credentials to send extension initialization request.
#### Extension Registration
In response to the REST `/initialize` request, `ExtensionsManager` discovers the extension listening on a predefined port and executes the TCP handshake protocol to
establish a data transfer connection. Then OpenSearch sends a request to the OpenSearch SDK for Java and, upon acknowledgment, the extension responds with its name. This
name is logged in the terminal where OpenSearch is running:
```bash
[2022-06-16T21:30:18,857][INFO ][o.o.t.TransportService ] [runTask-0] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
Expand All @@ -188,7 +180,8 @@ The OpenSearch SDK terminal also logs all requests and responses it receives fro
- TCP handshake request:
```bash
21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600, L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B
21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600,
L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
Expand All @@ -203,7 +196,8 @@ MESSAGE RECEIVED:E«󀀀internal:tcp/handshake£·A
- Extension name request/response:
```bash
21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B
21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b,
L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
Expand All @@ -221,66 +215,74 @@ MESSAGE RECEIVED:E«󀀀internal:tcp/handshake£·A
|000000b0| 65 6d 6f 74 65 5f 63 6c 75 73 74 65 72 5f 63 6c |emote_cluster_cl|
|000000c0| 69 65 6e 74 01 72 00 a3 8e b7 41 00 |ient.r....A. |
+--------+-------------------------------------------------+----------------+
MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX4bkVjG-Q 127.0.0.1 127.0.0.1 127.0.0.1´cluster_managermdatadingestiremote_cluster_clientr£·A
21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 204, request id: 3, type: request, version: 3.0.0, action: internal:discovery/extensions] READ: 204B
MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX4bkVjG-Q 127.0.0.1 127.0.0.1
127.0.0.1´cluster_managermdatadingestiremote_cluster_clientr£·A
21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532,
remoteAddress=/127.0.0.1:47782} [length: 204, request id: 3, type: request, version: 3.0.0, action: internal:discovery/extensions] READ: 204B
21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] received request
21:30:18.996 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - register 2 [transport] [internal:discovery/extensions] []
21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - unregister task for id: 2
21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 48, request id: 3, type: response, version: 3.0.0, header size: 2B] WRITE: 48B
21:30:18.998 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] WRITE: 48B
21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532,
remoteAddress=/127.0.0.1:47782} [length: 48, request id: 3, type: response, version: 3.0.0, header size: 2B] WRITE: 48B
21:30:18.998 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b,
L:/127.0.0.1:4532 - R:/127.0.0.1:47782] WRITE: 48B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 45 53 00 00 00 2a 00 00 00 00 00 00 00 03 01 08 |ES...*..........|
|00000010| 2d c7 23 00 00 00 02 00 00 16 4e 61 6d 65 64 57 |-.#.......exampl|
|00000020| 72 69 74 65 61 62 6c 65 52 65 67 69 73 74 72 79 |epluginname |
+--------+-------------------------------------------------+----------------+
21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] FLUSH
21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b,
L:/127.0.0.1:4532 - R:/127.0.0.1:47782] FLUSH
21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] sent response
```
It is important to ensure that the OpenSearch SDK for Java is already running on a separate process.
### Send a REST Request to the Extension
### Send a REST request to the extension
The following request is configured to be handled by the sample `HelloWorldExtension` (note that its matching `uniqueId` is `hello-world-java`):
The following request is configured to be handled by the sample `HelloWorldExtension` (note that its matching `uniqueId` is `opensearch-sdk-java-1`):
```bash
curl -X GET localhost:9200/_extensions/_opensearch-sdk-java-1/hello
curl -X GET localhost:9200/_extensions/_hello-world-java/hello
```
## Developing your own extension
## Developing Your Own Extension
Before you write your own extension, read through the [design documentation](DESIGN.md) to learn about extension architecture and class hierarchy. Then follow [this guide](CREATE_YOUR_FIRST_EXTENSION.md) to develop your own extension. For an example, see the sample Hello World extension in the `org.opensearch.sdk.sample.helloworld` package.
Before you write your own extension, read through the [design documentation](DESIGN.md) to learn about extension architecture and class hierarchy. Then follow [this
guide](CREATE_YOUR_FIRST_EXTENSION.md) to develop your own extension. For an example, see the sample Hello World extension in the `org.opensearch.sdk.sample.helloworld`
package.
Refer to the following sections for information about post-development tasks.
### Running a custom extension
### Running a Custom Extension
To run an extension that uses the SDK, use `./gradlew run` on that extension.
### Publishing the OpenSearch SDK for Java repo to the Maven local repo
### Publishing the OpenSearch SDK for Java Repo to Maven Local
Until we publish this repo to the Maven Central Repository, publishing to the Maven local repository is how extensions (outside of sample packages) import the artifacts:
```bash
./gradlew publishToMavenLocal
```
### Running tests
### Running Tests
Use the following command to run tests:
```
```bash
./gradlew clean build integTest
```
### Launching and debugging from an IDE
### Launching and Debugging from an IDE
For information about launching and debugging from an IDE in OpenSearch, see [this document](https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#launching-and-debugging-from-an-ide)
For information about launching and debugging from an IDE in OpenSearch, see [this
document](https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#launching-and-debugging-from-an-ide)
### Generating an artifact
### Generating an Artifact
In `opensearch-sdk-java`, navigate to `build/distributions`. Look for the tarball in the form `opensearch-sdk-java-1.0.0-SNAPSHOT.tar`. If there is no such tarball, use the following command to create one:
In `opensearch-sdk-java`, navigate to `build/distributions`. Look for the tarball in the form `opensearch-sdk-java-1.0.0-SNAPSHOT.tar`. If there is no such tarball, use
the following command to create one:
```bash
./gradlew clean && ./gradlew build
```
Expand All @@ -300,8 +302,11 @@ The artifact will include extension settings for the sample Hello World extensio
opensearchPort: 9200
```
You can optionally add `routeNamePrefix:` as a value to the yml. This setting allows you to prefix all your registered NamedRoute names.
The value must be alphanumeric and can contain `_` in the name.
Start the sample extension with `./bin/opensearch-sdk-java`
### Submitting changes
### Contributing Changes
To learn how to submit your changes, see [CONTRIBUTING](CONTRIBUTING.md).
2 changes: 1 addition & 1 deletion build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ buildscript {

plugins {
id 'java-library'
id "com.diffplug.spotless" version "6.21.0" apply false
id "com.diffplug.spotless" version "6.22.0" apply false
id 'jacoco'
id "com.form.diff-coverage" version "0.9.5"
// for javadocs and checks spotless doesn't do
Expand Down
19 changes: 19 additions & 0 deletions src/main/java/org/opensearch/sdk/sample/helloworld/hello.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"name": "Hello World",
"uniqueId": "hello-world-java",
"hostAddress": "127.0.0.1",
"port": "4500",
"version": "1.0",
"opensearchVersion": "3.0.0",
"minimumCompatibleVersion": "3.0.0",
"dependencies": [
{
"uniqueId": "test1",
"version": "2.0.0"
},
{
"uniqueId": "test2",
"version": "3.0.0"
}
]
}
Loading