Merge master0611 (#12703)

* [IOTDB-6336] Add max retry time duration and whether to retry for unknown errors configurations * Optimize error message when creating unsupported data type (#12650) Optimize error message when creating unsupported data type * Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters & Subscription: avoid poll messages from unsubscribed topics (#12648) * Pipe: fix uninitialized config when validate PipeHistoricalDataRegionTsFileExtractor parameters * Subscription: avoid poll messages from unsubscribed topics * Fix region migration's timeout judgement #12639 * fix compaction recover skip negative time partition (#12637) * Fix Concurrency Instability When IoTConsensus LogDispatcher Exits #12655 * JDBC: Fix IndexOutOfBoundsException when calling getBytes (#12662) * JDBC: Fix ArrayOutOfBoundException when calling getBytes * Update JDBCCharsetExample.java * [IOTDB-6335] Redundant rows when using GROUP BY TIME with LIMIT * Pipe: Enhance error handling logic in pipe async connector's handlers (#12669) * Print log when read resource error (#12666) * Load: Make load SQL option VERIFY can only be set to true (#12670) * Pipe: Improved hybrid algorithm and avoid the caculations of tsfiles in connector pending queue tsfiles to block the forwarding of realtime requests (#12668) * Migrate to Develocity Maven Extension (#12658) * Migrate to Develocity Maven Extension * Update to Common Custom User Data Maven Extension 2.0 * Update import schema logic implementation and optimize -help display information (#12643) * add import-schema.sh/bat and export-schema.sh/bat in cli/tools * fix import schema * fix import schema --------- Co-authored-by: 2b3c511 <[email protected]> * Subscription: fix topic now timestamp precision (#12663) * [IOTDB-6337] Refine the count calculation in RegionScan framework * Bump version of master branch to 1.3.3-SNAPSHOT (#12672) * Pipe: Reduce warn log when PipeConfigRegionExtractorMetrics does not exist (#12673) * [IOTDB-6061] Fix the instability failure caused by initServer in IoTConsensus UT not binding to the corresponding port (#12674) * fix concurrent * fix concurrent * Load: Make load SQL option VERIFY can only be set to true (A follow-up fix for #12670) (#12676) * Pipe: add compression level config for connector ZSTD compressor (#12630) Co-authored-by: Steve Yurong Su <[email protected]> * [PY-client] Unify setup.py and requirement.txt (#12601) * Pipe IT: Ignore failed cases caused by cluster restart failure (#12678) * update import and export data -help description (#12677) Co-authored-by: 2b3c511 <[email protected]> * Pipe: Fix potential NPE when lastEvent == null in PipeConnectorSubtask (#12680) * Fix wrong calculation for maxBytesCanReserve for IdentitySinkOperator * Change floating point encoding method in IT * Throw `IndexOutOfBound` exception in row implementation. * Thread safely SeriesPartitionTable (#12679) * finish * use concurrentskiplist * bug fix --------- Co-authored-by: OneSizeFitQuorum <[email protected]> * Pipe: Refactor some features for user access (#12686) * Set timestamp precision same as time column (#12681) * done * done * rollback change * fix IT * fix IT * Set configuration on node (#12626) Co-authored-by: Haonan <[email protected]> * Fix cli report error when ignoreTimestamp is true (#12691) * done * done * rollback change * fix IT * fix IT * done * invoke ci * done * invoke ci * Load: Add check for BufferUnderflowException (#12690) Co-authored-by: xz m <[email protected]> * remove excess stack * Fix new UDTF execution framework dead loop error. * Perfect methods of IAnalysis to adapt the write process of table model * add WriteFragmentParallelPlanner in TableModelDistributePlanner --------- Co-authored-by: Chen YZ <[email protected]> Co-authored-by: V_Galaxy <[email protected]> Co-authored-by: Li Yu Heng <[email protected]> Co-authored-by: shuwenwei <[email protected]> Co-authored-by: Xiangpeng Hu <[email protected]> Co-authored-by: Steve Yurong Su <[email protected]> Co-authored-by: YangCaiyin <[email protected]> Co-authored-by: Haonan <[email protected]> Co-authored-by: Caideyipi <[email protected]> Co-authored-by: Clay Johnson <[email protected]> Co-authored-by: Summer <[email protected]> Co-authored-by: 2b3c511 <[email protected]> Co-authored-by: Zikun Ma <[email protected]> Co-authored-by: Liao Lanyu <[email protected]> Co-authored-by: Zhihao Shen <[email protected]> Co-authored-by: Yongzao <[email protected]> Co-authored-by: OneSizeFitQuorum <[email protected]> Co-authored-by: ppppoooo <[email protected]> Co-authored-by: xz m <[email protected]> Co-authored-by: CritasWang <[email protected]> Co-authored-by: Beyyes <[email protected]>
apache · Jun 11, 2024 · d72429f · d72429f
1 parent 532e743
commit d72429f
Show file tree

Hide file tree

Showing 95 changed files with 1,765 additions and 382 deletions.
diff --git a/distribution/src/assembly/all.xml b/distribution/src/assembly/all.xml
@@ -57,6 +57,15 @@
         <fileSet>
             <outputDirectory>conf</outputDirectory>
             <directory>${maven.multiModuleProjectDirectory}/iotdb-core/node-commons/src/assembly/resources/conf</directory>
+            <excludes>
+                <exclude>**/*.sh</exclude>
+                <exclude>**/*.bat</exclude>
+                <exclude>**/iotdb-system.properties</exclude>
+            </excludes>
+        </fileSet>
+        <fileSet>
+            <outputDirectory>conf</outputDirectory>
+            <directory>${maven.multiModuleProjectDirectory}/iotdb-core/node-commons/target/conf</directory>
         </fileSet>
         <fileSet>
             <outputDirectory>sbin</outputDirectory>

diff --git a/distribution/src/assembly/confignode.xml b/distribution/src/assembly/confignode.xml
@@ -54,6 +54,15 @@
         <fileSet>
             <outputDirectory>conf</outputDirectory>
             <directory>${maven.multiModuleProjectDirectory}/iotdb-core/node-commons/src/assembly/resources/conf</directory>
+            <excludes>
+                <exclude>**/*.sh</exclude>
+                <exclude>**/*.bat</exclude>
+                <exclude>**/iotdb-system.properties</exclude>
+            </excludes>
+        </fileSet>
+        <fileSet>
+            <outputDirectory>conf</outputDirectory>
+            <directory>${maven.multiModuleProjectDirectory}/iotdb-core/node-commons/target/conf</directory>
         </fileSet>
         <!--    <fileSet>-->
         <!--      <directory>${maven.multiModuleProjectDirectory}/confignode/src/assembly/resources/tools</directory>-->

diff --git a/distribution/src/assembly/datanode.xml b/distribution/src/assembly/datanode.xml
@@ -45,6 +45,15 @@
         <fileSet>
             <outputDirectory>conf</outputDirectory>
             <directory>${maven.multiModuleProjectDirectory}/iotdb-core/node-commons/src/assembly/resources/conf</directory>
+            <excludes>
+                <exclude>**/*.sh</exclude>
+                <exclude>**/*.bat</exclude>
+                <exclude>**/iotdb-system.properties</exclude>
+            </excludes>
+        </fileSet>
+        <fileSet>
+            <outputDirectory>conf</outputDirectory>
+            <directory>${maven.multiModuleProjectDirectory}/iotdb-core/node-commons/target/conf</directory>
         </fileSet>
         <fileSet>
             <outputDirectory>sbin</outputDirectory>

diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBRepairDataIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBRepairDataIT.java
@@ -60,7 +60,7 @@ public static void setUp() throws Exception {
 
   @AfterClass
   public static void tearDown() throws Exception {
-    EnvFactory.getEnv().initClusterEnvironment();
+    EnvFactory.getEnv().cleanClusterEnvironment();
   }
 
   @Test

diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSetConfigurationIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSetConfigurationIT.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.it;
+
+import org.apache.iotdb.commons.conf.CommonConfig;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.env.cluster.node.ConfigNodeWrapper;
+import org.apache.iotdb.it.env.cluster.node.DataNodeWrapper;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.LocalStandaloneIT;
+
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.sql.Connection;
+import java.sql.Statement;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({LocalStandaloneIT.class})
+public class IoTDBSetConfigurationIT {
+  @BeforeClass
+  public static void setUp() throws Exception {
+    EnvFactory.getEnv().initClusterEnvironment();
+  }
+
+  @AfterClass
+  public static void tearDown() throws Exception {
+    EnvFactory.getEnv().cleanClusterEnvironment();
+  }
+
+  @Test
+  public void testSetConfiguration() throws Exception {
+    try (Connection connection = EnvFactory.getEnv().getConnection();
+        Statement statement = connection.createStatement()) {
+      statement.execute("set configuration \"enable_seq_space_compaction\"=\"false\"");
+      statement.execute("set configuration \"enable_unseq_space_compaction\"=\"false\" on 0");
+      statement.execute("set configuration \"enable_cross_space_compaction\"=\"false\" on 1");
+    } catch (Exception e) {
+      Assert.fail(e.getMessage());
+    }
+    for (ConfigNodeWrapper configNodeWrapper : EnvFactory.getEnv().getConfigNodeWrapperList()) {
+      String systemPropertiesPath =
+          configNodeWrapper.getNodePath()
+              + File.separator
+              + "conf"
+              + File.separator
+              + CommonConfig.SYSTEM_CONFIG_NAME;
+      File f = new File(systemPropertiesPath);
+      String content = new String(Files.readAllBytes(f.toPath()));
+      Assert.assertTrue(content.contains("enable_seq_space_compaction=false"));
+      Assert.assertTrue(content.contains("enable_unseq_space_compaction=false"));
+    }
+    for (DataNodeWrapper dataNodeWrapper : EnvFactory.getEnv().getDataNodeWrapperList()) {
+      String systemPropertiesPath =
+          dataNodeWrapper.getNodePath()
+              + File.separator
+              + "conf"
+              + File.separator
+              + CommonConfig.SYSTEM_CONFIG_NAME;
+      File f = new File(systemPropertiesPath);
+      String content = new String(Files.readAllBytes(f.toPath()));
+      Assert.assertTrue(content.contains("enable_seq_space_compaction=false"));
+      Assert.assertTrue(content.contains("enable_cross_space_compaction=false"));
+    }
+  }
+}
diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSettleIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSettleIT.java
@@ -56,7 +56,7 @@ public static void setUp() throws Exception {
   @AfterClass
   public static void tearDown() throws Exception {
     close();
-    EnvFactory.getEnv().initClusterEnvironment();
+    EnvFactory.getEnv().cleanClusterEnvironment();
   }
 
   @Test

diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSimpleQueryIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBSimpleQueryIT.java
@@ -39,7 +39,6 @@
 import java.sql.ResultSetMetaData;
 import java.sql.SQLException;
 import java.sql.Statement;
-import java.sql.Timestamp;
 import java.sql.Types;
 import java.time.LocalDate;
 import java.util.ArrayList;
@@ -1173,13 +1172,13 @@ public void testNewDataType() {
         while (resultSet.next()) {
           long time = resultSet.getLong(1);
           Date date = resultSet.getDate(2);
-          Timestamp timestamp = resultSet.getTimestamp(3);
+          long timestamp = resultSet.getLong(3);
           byte[] blob = resultSet.getBytes(4);
           String text = resultSet.getString(5);
           assertEquals(2024 - 1900, date.getYear());
           assertEquals(5 - 1, date.getMonth());
           assertEquals(time % 31 + 1, date.getDate());
-          assertEquals(time, timestamp.getTime());
+          assertEquals(time, timestamp);
           assertArrayEquals(byteArray, blob);
           assertEquals(String.valueOf(time), text);
         }

diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBAggregationIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/IoTDBAggregationIT.java
@@ -348,8 +348,8 @@ public void countTest() {
   public void firstTest() {
     String[] retArray =
         new String[] {
-          "0,2000,2000,2000.0,2000,2000,0x2000,2000-01-01,1970-01-01T00:00:02.000Z",
-          "0,500,500,500.0,500,500,0x0500,1500-01-01,1970-01-01T00:00:00.500Z"
+          "0,2000,2000,2000.0,2000,2000,0x2000,2000-01-01,2000",
+          "0,500,500,500.0,500,500,0x0500,1500-01-01,500"
         };
     try (Connection connection = EnvFactory.getEnv().getConnection();
         Statement statement = connection.createStatement()) {
@@ -453,9 +453,9 @@ public void firstTest() {
   public void lastTest() {
     String[] retArray =
         new String[] {
-          "0,8499,8499.0,8499,0x8499,8499-01-01,1970-01-01T00:00:08.499Z",
-          "0,1499,1499.0,1499,0x1499,1499-01-01,1970-01-01T00:00:01.499Z",
-          "0,2200,2200.0,2200,0x2200,2200-01-01,1970-01-01T00:00:02.200Z"
+          "0,8499,8499.0,8499,0x8499,8499-01-01,8499",
+          "0,1499,1499.0,1499,0x1499,1499-01-01,1499",
+          "0,2200,2200.0,2200,0x2200,2200-01-01,2200"
         };
     try (Connection connection = EnvFactory.getEnv().getConnection();
         Statement statement = connection.createStatement()) {
@@ -727,10 +727,7 @@ public void firstLastValueTest() throws SQLException {
   @Test
   public void maxminValueTest() {
     String[] retArray =
-        new String[] {
-          "0,8499,500.0,999,1000,8499-01-01,1000-01-01,1970-01-01T00:00:08.499Z,1970-01-01T00:00:00.500Z",
-          "0,2499,500.0"
-        };
+        new String[] {"0,8499,500.0,999,1000,8499-01-01,1000-01-01,8499,500", "0,2499,500.0"};
     try (Connection connection = EnvFactory.getEnv().getConnection();
         Statement statement = connection.createStatement()) {
 

diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/maxby/IoTDBMaxByIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/maxby/IoTDBMaxByIT.java
@@ -210,8 +210,7 @@ public void testMaxByWithDifferentXAndYInputTypes() {
               "root.db.d1",
               new String[] {"x1", "x2", "x3", "x4", "x5", "x6", "x7", "x8", "x9", "x10"},
               new String[] {"y1", "y2", "y3", "y4", "y7", "y9", "y10"});
-      String[] retArray =
-          new String[] {"3,3,3.0,3.0,false,3,3,0x33,2024-01-03,1970-01-01T00:00:00.003Z,"};
+      String[] retArray = new String[] {"3,3,3.0,3.0,false,3,3,0x33,2024-01-03,3,"};
       for (Map.Entry<String, String[]> expectedHeader : expectedHeaders.entrySet()) {
         String y = expectedHeader.getKey();
         resultSetEqualTest(

diff --git a/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/minby/IoTDBMinByIT.java b/integration-test/src/test/java/org/apache/iotdb/db/it/aggregation/minby/IoTDBMinByIT.java
@@ -210,8 +210,7 @@ public void testMaxByWithDifferentXAndYInputTypes() {
               "root.db.d1",
               new String[] {"x1", "x2", "x3", "x4", "x5", "x6", "x7", "x8", "x9", "x10"},
               new String[] {"y1", "y2", "y3", "y4", "y7", "y9", "y10"});
-      String[] retArray =
-          new String[] {"3,3,3.0,3.0,false,3,3,0x33,2024-01-03,1970-01-01T00:00:00.003Z,"};
+      String[] retArray = new String[] {"3,3,3.0,3.0,false,3,3,0x33,2024-01-03,3,"};
       for (Map.Entry<String, String[]> expectedHeader : expectedHeaders.entrySet()) {
         String y = expectedHeader.getKey();
         resultSetEqualTest(