-
Notifications
You must be signed in to change notification settings - Fork 1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor/ai node build update (#13349)
* Refactor/ai node build update (#13302) * fix: Hopefully fixed the build of the cpp-example on windows systems * fix: Increase the timeout even more to finally make the build succeed. * fix: Fix the build. * refactor: Cleaned up the build of the ai-node to be more robust * fix: Made the groovy script also work when building the entire project. * feat: Added some checks to validate the min and max version of the python3 version being used. * feat: Added some checks to validate the min and max version of the python3 version being used. * fix: Updated the python settings in pyproject.toml to reflect the really valid range of python versions and updated the prerequisite check accordingly. * feat: Removed the option to remove an AI node by rpc-ip+port. * chore: Added the poetry.lock to the .gitignore * docs: Removed the option to use the ip+port from the output of the scripts. * chore: Added the poetry.lock to the repo and removed it from the .gitignore file (cherry picked from commit 7c808c5) * set ainode version --------- Co-authored-by: Christofer Dutz <[email protected]>
- Loading branch information
Showing
9 changed files
with
1,882 additions
and
208 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,7 +21,7 @@ build-backend = "poetry.core.masonry.api" | |
|
|
||
| [tool.poetry] | ||
| name = "apache-iotdb-ainode" | ||
| version = "1.4.0.dev" | ||
| version = "1.3.3.dev" | ||
| description = "Apache IoTDB AINode" | ||
| readme = "README.md" | ||
| authors = ["Apache Software Foundation <[email protected]>"] | ||
|
|
@@ -46,7 +46,7 @@ packages = [ | |
| ] | ||
|
|
||
| [tool.poetry.dependencies] | ||
| python = "^3.8" | ||
| python = ">=3.8, <3.13" | ||
|
|
||
| numpy = "^1.21.4" | ||
| pandas = "^1.3.5" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| artifactId=iotdb-ainode | ||
| groupId=org.apache.iotdb | ||
| version=${project.version} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| import groovy.toml.TomlSlurper | ||
|
|
||
| import java.util.regex.Matcher | ||
|
|
||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | ||
| // The entire Python "check" block is borrowed from Apache PLC4X's build. | ||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | ||
|
|
||
| allConditionsMet = true | ||
|
|
||
| /** | ||
| * Version extraction function/macro. It looks for occurrence of x.y or x.y.z | ||
| * in passed input text (likely output from `program --version` command if found). | ||
| * | ||
| * @param input | ||
| * @return | ||
| */ | ||
| private static Matcher extractVersion(input) { | ||
| def matcher = input =~ /(\d+\.\d+(\.\d+)?).*/ | ||
| matcher | ||
| } | ||
|
|
||
| def checkVersionAtLeast(String current, String minimum) { | ||
| def currentSegments = current.tokenize('.') | ||
| def minimumSegments = minimum.tokenize('.') | ||
| def numSegments = Math.min(currentSegments.size(), minimumSegments.size()) | ||
| for (int i = 0; i < numSegments; ++i) { | ||
| def currentSegment = currentSegments[i].toInteger() | ||
| def minimumSegment = minimumSegments[i].toInteger() | ||
| if (currentSegment < minimumSegment) { | ||
| println current.padRight(14) + " FAILED (required min " + minimum + " but got " + current + ")" | ||
| return false | ||
| } else if (currentSegment > minimumSegment) { | ||
| println current.padRight(14) + " OK" | ||
| return true | ||
| } | ||
| } | ||
| def curNotShorter = currentSegments.size() >= minimumSegments.size() | ||
| if (curNotShorter) { | ||
| println current.padRight(14) + " OK" | ||
| } else { | ||
| println current.padRight(14) + " (required min " + minimum + " but got " + current + ")" | ||
| } | ||
| curNotShorter | ||
| } | ||
|
|
||
| def checkVersionAtMost(String current, String maximum) { | ||
| def currentSegments = current.tokenize('.') | ||
| def maximumSegments = maximum.tokenize('.') | ||
| def numSegments = Math.min(currentSegments.size(), maximumSegments.size()) | ||
| for (int i = 0; i < numSegments; ++i) { | ||
| def currentSegment = currentSegments[i].toInteger() | ||
| def maximumSegment = maximumSegments[i].toInteger() | ||
| if (currentSegment > maximumSegment) { | ||
| println current.padRight(14) + " FAILED (required max " + maximum + " but got " + current + ")" | ||
| return false | ||
| } else if (currentSegment < maximumSegment) { | ||
| println current.padRight(14) + " OK" | ||
| return true | ||
| } | ||
| } | ||
| def curNotShorter = currentSegments.size() >= maximumSegments.size() | ||
| if (curNotShorter) { | ||
| println current.padRight(14) + " OK" | ||
| } else { | ||
| println current.padRight(14) + " (required max " + maximum + " but got " + current + ")" | ||
| } | ||
| curNotShorter | ||
| } | ||
|
|
||
| def checkPython() { | ||
| String python = project.properties['python.exe.bin'] | ||
| println "Using python executable: " + python.padRight(14) + " OK" | ||
| print "Detecting Python version: " | ||
| try { | ||
| def process = (python + " --version").execute() | ||
| def stdOut = new StringBuilder() | ||
| def stdErr = new StringBuilder() | ||
| process.waitForProcessOutput(stdOut, stdErr) | ||
| Matcher matcher = extractVersion(stdOut + stdErr) | ||
| if (matcher.size() > 0) { | ||
| String curVersion = matcher[0][1] | ||
| def result = checkVersionAtLeast(curVersion, "3.8.0") | ||
| if (!result) { | ||
| allConditionsMet = false | ||
| } | ||
| result = checkVersionAtMost(curVersion, "3.13") | ||
| if (!result) { | ||
| allConditionsMet = false | ||
| } | ||
| } else { | ||
| println "missing (Please install at least version 3.8.0 and at most one of the 3.13.x versions)" | ||
| allConditionsMet = false | ||
| } | ||
| } catch (Exception ignored) { | ||
| println "missing" | ||
| println "--- output of version `${python} --version` command ---" | ||
| println output | ||
| println "----------------------------------------------------" | ||
| allConditionsMet = false | ||
| } | ||
| } | ||
|
|
||
|
|
||
| // On Ubuntu it seems that venv is generally available, but the 'ensurepip' command fails. | ||
| // In this case we need to install the python3-venv package. Unfortunately checking the | ||
| // venv is successful in this case, so we need this slightly odd test. | ||
| def checkPythonVenv() { | ||
| print "Detecting venv: " | ||
| try { | ||
| def python = project.properties['python.exe.bin'] | ||
| def cmdArray = [python, "-Im", "ensurepip"] | ||
| def process = cmdArray.execute() | ||
| def stdOut = new StringBuilder() | ||
| def stdErr = new StringBuilder() | ||
| process.waitForProcessOutput(stdOut, stdErr) | ||
| if (stdErr.contains("No module named")) { | ||
| println "missing" | ||
| println "--- output of version `python -Im \"ensurepip\"` command ---" | ||
| println output | ||
| println "------------------------------------------------------------" | ||
| allConditionsMet = false | ||
| } else { | ||
| println " OK" | ||
| } | ||
| } catch (Exception e) { | ||
| println "missing" | ||
| println "--- failed with exception ---" | ||
| println e | ||
| e.printStackTrace() | ||
| println "----------------------------------------------------" | ||
| allConditionsMet = false | ||
| } | ||
| } | ||
|
|
||
| // Check the python environment is setup correctly. | ||
| checkPython() | ||
| checkPythonVenv() | ||
|
|
||
| if (!allConditionsMet) { | ||
| throw new RuntimeException("Not all conditions met, see log for details.") | ||
| } | ||
| println "" | ||
| println "All known conditions met successfully." | ||
| println "" | ||
|
|
||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | ||
| // Calculate the version that we should use in the python build. | ||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | ||
|
|
||
| def currentMavenVersion = project.version as String | ||
| def currentPythonVersion = currentMavenVersion | ||
| if(currentMavenVersion.contains("-SNAPSHOT")) { | ||
| currentPythonVersion = currentMavenVersion.split("-SNAPSHOT")[0] + ".dev" | ||
| } | ||
| println "Current Project Version in Maven: " + currentMavenVersion | ||
| println "Current Project Version in Python: " + currentPythonVersion | ||
|
|
||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | ||
| // Synchronize the version in pyproject.toml and the one used in the maven pom. | ||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | ||
|
|
||
| def pyprojectFile = new File(project.basedir, "pyproject.toml") | ||
| def ts = new TomlSlurper() | ||
| def toml = ts.parse(pyprojectFile) | ||
| def pyprojectFileVersion = toml.tool.poetry.version | ||
| if (pyprojectFileVersion != currentPythonVersion) { | ||
| pyprojectFile.text = pyprojectFile.text.replace("version = \"" + pyprojectFileVersion + "\"", "version = \"" + currentPythonVersion + "\"") | ||
| println "Version in pyproject.toml updated from " + pyprojectFileVersion + " to " + currentPythonVersion | ||
| // TODO: When releasing, we might need to manually add this file to the release preparation commit. | ||
| } else { | ||
| println "Version in pyproject.toml is up to date" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters