forked from apache/druid
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
support Aliyun OSS service as deep storage (apache#9898)
* init commit, all tests passed * fix format Signed-off-by: frank chen <[email protected]> * data stored successfully * modify config path * add doc * add aliyun-oss extension to project * remove descriptor deletion code to avoid warning message output by aliyun client * fix warnings reported by lgtm-com * fix ci warnings Signed-off-by: frank chen <[email protected]> * fix errors reported by intellj inspection check Signed-off-by: frank chen <[email protected]> * fix doc spelling check Signed-off-by: frank chen <[email protected]> * fix dependency warnings reported by ci Signed-off-by: frank chen <[email protected]> * fix warnings reported by CI Signed-off-by: frank chen <[email protected]> * add package configuration to support showing extension info Signed-off-by: frank chen <[email protected]> * add IT test cases and fix bugs Signed-off-by: frank chen <[email protected]> * 1. code review comments adopted 2. change schema from 'aliyun-oss' to 'oss' Signed-off-by: frank chen <[email protected]> * add license info Signed-off-by: frank chen <[email protected]> * fix doc Signed-off-by: frank chen <[email protected]> * exclude execution of IT testcases of OSS extension from CI Signed-off-by: frank chen <[email protected]> * put the extensions under contrib group and add to distribution * fix names in test cases * add unit test to cover OssInputSource * fix names in test cases * fix dependency problem reported by CI Signed-off-by: frank chen <[email protected]>
- Loading branch information
1 parent
e2c5bcc
commit 60c6bd5
Showing
45 changed files
with
5,890 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
54 changes: 54 additions & 0 deletions
54
docs/development/extensions-contrib/aliyun-oss-extensions.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| --- | ||
| id: aliyun-oss | ||
| title: "Aliyun OSS" | ||
| --- | ||
|
|
||
| <!-- | ||
| ~ Licensed to the Apache Software Foundation (ASF) under one | ||
| ~ or more contributor license agreements. See the NOTICE file | ||
| ~ distributed with this work for additional information | ||
| ~ regarding copyright ownership. The ASF licenses this file | ||
| ~ to you under the Apache License, Version 2.0 (the | ||
| ~ "License"); you may not use this file except in compliance | ||
| ~ with the License. You may obtain a copy of the License at | ||
| ~ | ||
| ~ http://www.apache.org/licenses/LICENSE-2.0 | ||
| ~ | ||
| ~ Unless required by applicable law or agreed to in writing, | ||
| ~ software distributed under the License is distributed on an | ||
| ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| ~ KIND, either express or implied. See the License for the | ||
| ~ specific language governing permissions and limitations | ||
| ~ under the License. | ||
| --> | ||
|
|
||
|
|
||
| To use this Apache Druid extension, make sure to [include](../../development/extensions.md#loading-extensions) `aliyun-oss-extensions` extension. | ||
|
|
||
| ## Deep Storage | ||
|
|
||
| [Aliyun](https://www.aliyun.com) is the 3rd largest cloud infrastructure provider in the world. It provides its own storage solution known as OSS, [Object Storage Service](https://www.aliyun.com/product/oss). | ||
|
|
||
| To use aliyun OSS as deep storage, first config as below | ||
|
|
||
| |Property|Description|Possible Values|Default| | ||
| |--------|---------------|-----------|-------| | ||
| |`druid.oss.accessKey`|the `AccessKey ID` of your account which can be used to access the bucket| |Must be set.| | ||
| |`druid.oss.secretKey`|the `AccessKey Secret` of your account which can be used to access the bucket| |Must be set. | | ||
| |`druid.oss.endpoint`|the endpoint url of your OSS storage| |Must be set.| | ||
|
|
||
| if you want to use OSS as deep storage, use the configurations below | ||
|
|
||
| |Property|Description|Possible Values|Default| | ||
| |--------|---------------|-----------|-------| | ||
| |`druid.storage.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.| | ||
| |`druid.storage.oss.bucket`|storage bucket name.| | Must be set.| | ||
| |`druid.storage.oss.prefix`|a prefix string prepended to the file names for the segments published to aliyun OSS deep storage| druid/segments | | | ||
|
|
||
| To save index logs to OSS, apply the configurations below: | ||
|
|
||
| |Property|Description|Possible Values|Default| | ||
| |--------|---------------|-----------|-------| | ||
| |`druid.indexer.logs.type`| Global deep storage provider. Must be set to `oss` to make use of this extension. | oss |Must be set.| | ||
| |`druid.indexer.logs.oss.bucket`|the bucket used to keep logs| |Must be set.| | ||
| |`druid.indexer.logs.oss.prefix`|a prefix string prepended to the log files.| | | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <!-- | ||
| ~ Licensed to the Apache Software Foundation (ASF) under one | ||
| ~ or more contributor license agreements. See the NOTICE file | ||
| ~ distributed with this work for additional information | ||
| ~ regarding copyright ownership. The ASF licenses this file | ||
| ~ to you under the Apache License, Version 2.0 (the | ||
| ~ "License"); you may not use this file except in compliance | ||
| ~ with the License. You may obtain a copy of the License at | ||
| ~ | ||
| ~ http://www.apache.org/licenses/LICENSE-2.0 | ||
| ~ | ||
| ~ Unless required by applicable law or agreed to in writing, | ||
| ~ software distributed under the License is distributed on an | ||
| ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| ~ KIND, either express or implied. See the License for the | ||
| ~ specific language governing permissions and limitations | ||
| ~ under the License. | ||
| --> | ||
|
|
||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <groupId>org.apache.druid.extensions.contrib</groupId> | ||
| <artifactId>aliyun-oss-extensions</artifactId> | ||
| <name>aliyun-oss-extensions</name> | ||
| <description>aliyun-oss-extensions</description> | ||
|
|
||
| <parent> | ||
| <groupId>org.apache.druid</groupId> | ||
| <artifactId>druid</artifactId> | ||
| <version>0.19.0-SNAPSHOT</version> | ||
| <relativePath>../../pom.xml</relativePath> | ||
| </parent> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>org.apache.druid</groupId> | ||
| <artifactId>druid-core</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> <!-- aliyun oss --> | ||
| <groupId>com.aliyun.oss</groupId> | ||
| <artifactId>aliyun-sdk-oss</artifactId> | ||
| <version>3.3.0</version> | ||
| </dependency> | ||
|
|
||
| <dependency> | ||
| <groupId>com.fasterxml.jackson.module</groupId> | ||
| <artifactId>jackson-module-guice</artifactId> | ||
| <version>${jackson.version}</version> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>commons-io</groupId> | ||
| <artifactId>commons-io</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.fasterxml.jackson.core</groupId> | ||
| <artifactId>jackson-annotations</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>joda-time</groupId> | ||
| <artifactId>joda-time</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.google.inject</groupId> | ||
| <artifactId>guice</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.fasterxml.jackson.core</groupId> | ||
| <artifactId>jackson-databind</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.fasterxml.jackson.core</groupId> | ||
| <artifactId>jackson-core</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.google.inject.extensions</groupId> | ||
| <artifactId>guice-multibindings</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.google.guava</groupId> | ||
| <artifactId>guava</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>javax.validation</groupId> | ||
| <artifactId>validation-api</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.google.code.findbugs</groupId> | ||
| <artifactId>jsr305</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>commons-lang</groupId> | ||
| <artifactId>commons-lang</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
|
|
||
| <!-- Tests --> | ||
| <dependency> | ||
| <groupId>org.apache.druid</groupId> | ||
| <artifactId>druid-core</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <type>test-jar</type> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>junit</groupId> | ||
| <artifactId>junit</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.druid</groupId> | ||
| <artifactId>druid-server</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.druid</groupId> | ||
| <artifactId>druid-processing</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.easymock</groupId> | ||
| <artifactId>easymock</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>nl.jqno.equalsverifier</groupId> | ||
| <artifactId>equalsverifier</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.hamcrest</groupId> | ||
| <artifactId>hamcrest-core</artifactId> | ||
| <version>1.3</version> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
| <build> | ||
| <plugins> | ||
| <plugin><!--package all dependencies together--> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-assembly-plugin</artifactId> | ||
| <configuration> | ||
| <appendAssemblyId>false</appendAssemblyId> | ||
| <descriptorRefs> | ||
| <descriptorRef>jar-with-dependencies</descriptorRef> | ||
| </descriptorRefs> | ||
| <archive><!--add entries like Implementation-Title to MANIFEST.MF--> | ||
| <manifest> | ||
| <addDefaultImplementationEntries>true</addDefaultImplementationEntries> | ||
| </manifest> | ||
| </archive> | ||
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <id>make-assembly</id> | ||
| <phase>package</phase> | ||
| <goals> | ||
| <goal>single</goal> | ||
| </goals> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
| </project> |
Oops, something went wrong.