Vcpkg support for Bloom #541
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Notice that the `get_package_data` method will return a list. so it's clear that the logic `detect_branches` method in ReleaseGenerator and RosReleaseGenerator is same. In this commit, I reuse this function and put it into ReleaseGenerator's `handle_arguments` method so that we can safely delete the duplicate part in RosReleaseGenerator.
We need to put the `missing_dep_resolver` in class as staticmethod in case successor of Generator want to reuse this function. And I also add os_name, os_versino, ros_distro if oneday the resolver need more dependency information.
Firstly put the common place/process template files to common.py. Then, since every pacakge_system has its prefer way of formatting description and depends, so pass them as function. Next, rewrite get_sub_hook in DebianGenerator for debian specific substitutions attribute, and consider generate_cmd will use the hook, I make the hook as staticmethod method. At last, fix up the api change influence.
The major different part of RPM and Debian's generator as follow: 1. rpm don't have native option in local generate_cmd related command 2. rpm and debian's Date, changelogs, and License format are different 3. rpm need NoArch substitution 4. rpm has its description and depends specification And I also update the utils function like __process_template_folder, __place_template_folder, and place_tempalte_folder, the newest code in debian should also fit rpm generator's logic
Also fix typo here(unvalid => invalid)
This commit add the missed rosdistro when call super's get_subs_hook in rosdebian or rosrpm. The release_history should be passed as **kwargs format since it have default value already. And I also combine two line in get_release_tag method since it's not exceed 120 lines.
Neither Fedora or RHEL 7+ use this value. If another distribution were to be supported for RPM generation in Bloom that can take advantage of this value, we should find a better way to add it instead of hard- coding it into the template. Signed-off-by: Scott K Logan <[email protected]>
Supported in RPM 4.11 and newer: https://rpm.org/user_doc/autosetup.html Signed-off-by: Scott K Logan <[email protected]>
...rather than assuming that the make executable is 'make'. Signed-off-by: Scott K Logan <[email protected]>
Also move terminating '..' to the end to make it easier to patch cmake arguments and less likely that the make arguments accidentally get added to the end of the cmake arguments. Signed-off-by: Scott K Logan <[email protected]>
There are two classes of issues here: 1. ROS packages often provide libraries which are also provided by the operating system. If an operating system declares a dependency on that library, we don't want the package manager to install the ROS package instead of the system package. 2. Many ROS packages don't install libraries in a way that the 'provides' portion of dependency generation can detect, so when another ROS package takes a dependency on that library, the automatic dependency can't be met and the downstream package cannot be installed. More info on RPM dependency generation: https://rpm.org/user_doc/dependency_generators.html Signed-off-by: Scott K Logan <[email protected]>
The BRP Python bytecompiler will always use the sytem's default Python interpreter, which may not be the interpreter we're targeting. Safest option is to disable the automagic byte compilation altogether. Note that this doesn't mean that python files won't ever be compiled, it just means that the catch-all policy at the end of the process won't attempt to compile anything which hasn't already been compiled. Signed-off-by: Scott K Logan <[email protected]>
This change shouldn't modify the behavior in Fedora, where all current releases define 'cmake3' to be the same as 'cmake'. In RHEL 7, where cmake 2 is the default, we need to use the 'cmake3' macro to use the supplamental 'cmake3' executable instead of the system default. All current ROS releases except Indigo have a *minimum* cmake requirement of 3. Signed-off-by: Scott K Logan <[email protected]>
…e#540) The SCM plugin for mock will create an archive of the sources if either this marker file is present in the root of the project or the `write_tar` option is specified in mock's configuration. The way bloom works, we always want to archive the sources, but this isn't the default configuration of the SCM plugin. Writing this marker file means one less configuration is necessary when consuming the release repo to build the RPMs with mock. Signed-off-by: Scott K Logan <[email protected]>
Since this method won't use self's arrtibutes, change it in case someone need `tag name` during another staticmethod in the future.
Since some packages may have python, ruby related dependencies, not just pure C/C++ dependencies, so I leave the Build-Depends to empty, hope we can handle the dependencies outside vcpkg. And I think we could use Chocolatey to handle the ROS dependencies installation
- We need set VCPKG_BUILD_TYPE as `release` otherwise vcpkg will build `debug` & `release` version together - We will download Bloom released pacakge from repository via `vcpkg_from_<source_sites>` - We don't set `CMAKE_INSTALL_PREFIX` to OPTIONS, since vcpkg will set it to <CURRENT_PACKAGES_DIR>, and that's enough for us to pack. - We can't use `CMAKE_PREFIX_PATH` anymore since vcpkg will change it, use `AMENT_PREFIX_PATH` instead
- check python did exists, otherwise complain - follow vcpkg's style, set install path to <CURRENT_PACKAGES_DIR>. - make the install directory first, otherwise the cmake will complain
- this four templates is same - we decide the nuspec's metadata during Bloom's generation - we only handle files in `tools` directory, so the built binary files should be compressed and move to tools directory
- we decide the packages' build dependencies in Bloom's generation - the build_dependencies.config is used for installing packages dependencies via `choco install build_dependencies.config`, because vcpkg could install all ROS packages' dependencies.
The changes adapter from old generator including: - copy the format_description from DebianGenerator, this formatted description will be put in nuspec metadata's description part - format depends I modify format_depends function from DebianGenerator, change the versions dict adapter NuGet style, as you can see in: https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards - notice that `prepare_arguments` & `handle_arguments` call the parent's method in end of the function, which follow the Bloom's style, - the commit style in `generate_package` is similar to RPMGenerator The special changes in this commit - we don't handle release history in `generate_package`, since NuGet or Vcpkg don't require it
this generate cmd is similar to debian or rpm's generate_cmd
this generator follow the style of old ros generators But notice that I call VcpkgGenerator.get_subs_hook in the end of RosVcpkgGenerator's get_subs_hook, this way follow the Bloom style. If we don't do this way, the appended BuildDepends and Depends won't be updated to VcpkgGenerator.get_subs_hook, and will cause depend losed problem.
|
This looks reasonably mature, are there any plans to merge this PR anytime soon? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Abstract
This PR add VcpkgGenerator, RosVcpkgGenerator, and vcpkg's local generator, we can use these generators to expand the templates in vcpkg template folder and get vcpkg's CONTROL, portfiles.cmake; Chocolatey's pkg-name.nuspace, build_dependencies.config. And I validate the generated files in Windows platform.
And notice that this PR use the refactor work in the PR, and you can review code started from the commit message starts with
[vcpkg]if you have checked that PR before.Motivation
Now Bloom could generate Debian & RPM's meta content for ros_buildfarm to compile the ROS package and pack binary files then distribute it. Though there are some works target on other platform, like Arch support: 1, 2, these work haven't made use of the power of ros_buildfarm, and the users in other platforms need to compile ROS themselves, or download the official pre-built ROS archive, it takes lots of time. We all hope that ROS could be used on other platforms easier.
On the other side, the
vcpkgis a cross-platform C/C++ package manager, if we could usevcpkgto compile the ROS package, we may utilize it to handle ROS's pre-built process in the platformsvcpkgsupport. The following part is the details of my lab with my mentors: @cottsay @nuclearsandwich, and with the help of @seanyen @traversaro.Details
Generator part
The generator is used for analyzing the meta-data in ROS packages' package.xml, and resolve the package's dependencies in its target platform. But the rosdistro work for Windows is still WIP. So we need to do some hack now to expand the template files. What I do including:
For now, we need these hacks to test the vcpkg generators.
Template part
The templates in vcpkg folder include four-part:
CONTROL.em: this is used for vcpkg's dependencies analyze, but we don't add the dependencies inCONTROL.emsince our ROS package have extra Python dependencies, so thevcpkgC/C++ package manager couldn't handle it, let Chocolatey do this part job.portsfile.cmake.em: this is used for vcpkg building process. In general, we download the certain tag archive from ROS package's releasing repository, then use cmake or python setup.py to install the binary files to vcpkg's<CURRENT_PACKAGES_DIR>folder. Then we can pack the files.template.nuspec.em: this is used to collect the meta-data from ROS package's pacakge.xml. Then NuGet will analyze the dependencies(aka execute depends) in this file, and install the dependencies package for us.build_dependencies.em: since NuGet cannot specify build depends and execute depends, we need an extra build dependencies files for pre-install dependencies before build package.For example, you can see the expanded template
Validation
Now we can use vcpkg to build the ROS2 packages for ROS2 ports and export the built binary files to Chocolatey packages. The releasing files include ros_base, rviz Chocolatey packages, and its dependencies. You can download via https://github.com/lennonwoo/vcpkg/releases/tag/v0.11
If you have done the ROS2 build prerequisites. You can download the Chocolatey packages archive and uncompress it to your local device. Then use command
choco install ros-dashing-ros-base ros-dashing-demo-nodes-py ros-dashing-demo-nodes-cpp ros-dashing-demo-nodes-cpp-native -y -s <Your uncompress location>to install the
ros_basemeta-package and simple demo. The packages will be installed toC:\opt\ros\dashing, and then you can do a simple testBuild details
I use Bloom to generate 90 meta-packages and its 240 sub-packages'
vcpkgport files, as you can see in my forked repostory Then I use a build.py executed in administrator mode tochoco install build_dependencies.configinstall.bat <pkg-name>We also need to change something in vcpkg for building the ROS2 packages successfully, and you can see the detail commits
VCPKG_DEFAULT_TRIPLETenvironment variable, which value should be "x64-windows"For Ros2's package, there are somethings we need to patch:
That's all, thank you!