From 1341b6899ab12ae0f4ca7ef6df206a1c954bb529 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Fri, 25 Aug 2017 16:23:08 -0700 Subject: [PATCH 1/6] add article about dependency groups --- articles/102_dependency_groups.md | 109 ++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 articles/102_dependency_groups.md diff --git a/articles/102_dependency_groups.md b/articles/102_dependency_groups.md new file mode 100644 index 000000000..94d8e0d12 --- /dev/null +++ b/articles/102_dependency_groups.md @@ -0,0 +1,109 @@ +--- +layout: default +title: Dependency groups +permalink: articles/dependency_groups.html +abstract: + This article describes the concept of dependency groups, their use cases, how they can described in the package manifest, and how they can be mapped into binary packages. +author: '[Dirk Thomas](https://github.com/dirk-thomas)' +published: true +categories: Overview +--- + +{:toc} + +# {{ page.title }} + +
+{{ page.abstract }} +
+ +Original Author: {{ page.author }} + +## Preface + +In the ROS ecosystem a ROS package provides information about its dependencies in the manifest. +The latest format is specified in the [REP 140](http://www.ros.org/reps/rep-0140.html). +It supports different kind of dependencies (`build`, `run`, `test`, `doc`, etc.). +But each dependency is mandatory and needs to satisfied on its own. + +This article describes the concept of "dependency groups" which could be used for different purposes. +In terms of terminology a dependency group is just a set of dependencies. +For such a group a different semantic can then be applied (instead of just satisfy each dependency on its own). + +## Use Cases + +The following subsections will describe two different use cases how dependencies could be used. + +### (1) Need at least one "Provider" + +A package might want to declare that at least one dependency from a group needs to be present. +Compared to "normal" dependencies it is not requires that all dependencies of that group need to be present to satisfy the dependency group. + +An example for this is the `rmw_implementation` package. +Currently it depends on all RMW implementation packages in order to ensure that it may only be build after all RMW implementation packages have been built. +The goal would be that at least one RMW implementation needs to be present in order to build or use the `rmw_implementation` package. +It would be acceptable if more or even all RMW implementations are available but that is not required. + +### (2) All packages of a specific "type" + +Another use case is that a dependency group doesn't need to be explicitly defined by all dependency names. +Instead a dependency group is defined by an identifier and the expectation is that all packages which have that identifier in their manifest should be available. + +An example for this is the `ros1_bridge` package. +Currently it depends on a manually selected set of message packages. +The goal would be that it can declare that it depends on *all* packages which provide messages. + +## Top Down + +First, we look at the capabilities of binary packages if and how these use cases can be satisfied since we don't have much influence on these capabilities and have to use the features available. +Additionally we consider the needs for a from-source build. +Since ROS provides its own build tools it should be possible to adjust the logic to whatever is necessary to support these features. +Last, we define the information we need to additionally provide in the package manifest and how to integrate them into the existing structure. + +### Binary Packages + +For know this article focuses on Debian packages. +It would be great if someone could add the similar information for other platforms to ensure the concept is solid and works across mutliple platforms. + +#### Binary Packages + +... + +### Building from Source + +... + +### Information in the Manifest + +To provide the information for the build tool as well as the packaging process the following information need to be provided by the manifest: + +* Use case (1) + + * In a specific package a set of dependencies need to be associated to a dependency group. + * The semantic of the group needs to be defined (in this case "at-least-one-from-the-group"). + +* Use case (2) + + * A package declares a dependency on a group identifier. + * A set of packages declares that group identifier. + +#### Possible Format Extensions + +... + +## Proposed Solution + +### Extensions for the Package Manifest + +... + +### Build Tool + +For use case (1) the build tool only needs to treat all dependencies of the set as "normal" dependencies. +When they are available within the workspace they should be processed before and being made available to the package declaring the dependency group. + +For use case (2) the build tool needs to extract the identifiers from each package and consider them when detemining the topological order. + +### Packaging process + +The packaging tool needs to map the dependencies groups and identifiers to the platform specific files as described in the above section. From dd1d0b46464c2592684061478fa946cdae28f803 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 10 Oct 2017 15:46:39 -0700 Subject: [PATCH 2/6] add concrete information and possible format extensions --- articles/102_dependency_groups.md | 127 +++++++++++++++++++++--------- 1 file changed, 91 insertions(+), 36 deletions(-) diff --git a/articles/102_dependency_groups.md b/articles/102_dependency_groups.md index 94d8e0d12..0b25e284c 100644 --- a/articles/102_dependency_groups.md +++ b/articles/102_dependency_groups.md @@ -23,11 +23,11 @@ Original Author: {{ page.author }} In the ROS ecosystem a ROS package provides information about its dependencies in the manifest. The latest format is specified in the [REP 140](http://www.ros.org/reps/rep-0140.html). -It supports different kind of dependencies (`build`, `run`, `test`, `doc`, etc.). +It supports different kind of dependencies (`build`, `exec`, `test`, `doc`, etc.). But each dependency is mandatory and needs to satisfied on its own. This article describes the concept of "dependency groups" which could be used for different purposes. -In terms of terminology a dependency group is just a set of dependencies. +In terms of terminology a dependency group is a set of dependencies identified by the name of the group. For such a group a different semantic can then be applied (instead of just satisfy each dependency on its own). ## Use Cases @@ -36,7 +36,7 @@ The following subsections will describe two different use cases how dependencies ### (1) Need at least one "Provider" -A package might want to declare that at least one dependency from a group needs to be present. +A package might want to declare that at least one dependency from a group needs to be present (called `satisfy-at-least-one`). Compared to "normal" dependencies it is not requires that all dependencies of that group need to be present to satisfy the dependency group. An example for this is the `rmw_implementation` package. @@ -44,66 +44,121 @@ Currently it depends on all RMW implementation packages in order to ensure that The goal would be that at least one RMW implementation needs to be present in order to build or use the `rmw_implementation` package. It would be acceptable if more or even all RMW implementations are available but that is not required. +In case multiple package declare to be part of that dependency group the `rmw_implementation` package might want to provide an order list of preferred names which should be considered in that order. + ### (2) All packages of a specific "type" -Another use case is that a dependency group doesn't need to be explicitly defined by all dependency names. -Instead a dependency group is defined by an identifier and the expectation is that all packages which have that identifier in their manifest should be available. +Another use case is that a dependency group doesn't need to be explicitly defined by all dependency names (called `satisfy-all`). +Instead a dependency group is defined by an identifier and the expectation is that all packages which declare that they are being part of that group should be available. An example for this is the `ros1_bridge` package. Currently it depends on a manually selected set of message packages. The goal would be that it can declare that it depends on *all* packages which provide messages. -## Top Down +Since the `ros1_bridge` manifest only contains the name of the group dependency in doesn't control which packages declare that they are part of that group. +It might want to exclude specific packages to be considered even though they are in the group. + +## Processes + +The following two processes need to be update to consider dependency groups. + +### Build tool + +The build tool needs to consider the declared dependency groups when computing the topological order. +Since it operates on a set packages in a workspace it has all manifests for all of these packages available. +Independent of the semantic of the dependency group (satisfy-all vs. satisfy-at-least-one) the build tool can simply expand a dependency group to all packages of the set. + +### Release tool + +Since the release tool `bloom` operates on a single package it can't resolve a dependency group without further information. +It will need all package manifests available from `rosdistro` to resolve a dependency group to a set of packages. + +For the `satisfy-all` semantic it can simply generate dependencies on all packages from the group. +For the `satisfy-at-least-one` semantic the release tool could either use a platform specific mechanism like `provides` in Debian control files or simply expand the group to all packages (same as `satisfy-all`). +This design document doesn't aim to specify what the "best" behavior for the release tool is since the decision is likely platform dependent. + +Since the release tool expands the dependency groups when being invoked packages which join the group after that release won't be used until the package defining the group dependency is being re-released. + +### rosinstall_generator + +The `rosinstall_generator` already utilized all manifests from the `rosdistro` to decide which packages / repositories should be fetched. +This design document doesn't aim to specify what the "best" behavior for this tool is. +It could either match the behavior of the build tool or of the release tool or even something different. + +## Information in the Manifest + +To provide the necessary metadata for the build and release tools the following information must be provided by the manifest: + +* (A) A specific package declares a group dependency and uses a name to identify the group. + The dependency can be of any type specified in [REP 140](http://www.ros.org/reps/rep-0140.html): e.g. `build`, `exec`, `test`, `doc`. -First, we look at the capabilities of binary packages if and how these use cases can be satisfied since we don't have much influence on these capabilities and have to use the features available. -Additionally we consider the needs for a from-source build. -Since ROS provides its own build tools it should be possible to adjust the logic to whatever is necessary to support these features. -Last, we define the information we need to additionally provide in the package manifest and how to integrate them into the existing structure. +* (B) The semantic of the group dependency needs to be defined (in case (1) `satisfy-at-least-one`, in case (2) `satify-all`). -### Binary Packages +* (C) A list of preferred / excluded packages can be enumerated for each dependency group. -For know this article focuses on Debian packages. -It would be great if someone could add the similar information for other platforms to ensure the concept is solid and works across mutliple platforms. +* (D) Any package can declare that it is part of that group. -#### Binary Packages +### Possible Format Extensions -... +The group dependency (A) could be specified in two ways: -### Building from Source +* (A.1) Using the existing `*_depend` tags and adding an attribute to the tag to make it a group dependency. +* (A.2) Defining new `*_group_depend` tags which specifically identify group dependencies. -... +Considering the existing API in `catkin_pkg` which exposes the different dependencies as members of the `Package` class (A.1) would require existing code to distinguish the type of the dependency (group vs. no-group). +On the other hand (A.2) will simply add additional members which doesn't require existing code to change until it wants to support handling group dependencies. -### Information in the Manifest +The semantic of the group dependency (B) can be modeled as an attribute of the group dependency tag. -To provide the information for the build tool as well as the packaging process the following information need to be provided by the manifest: +Each preferred / excluded package (C) name can be listed in child-tags of the dependency group to keep the information atomic. -* Use case (1) +The group membership (D) could be specified in two ways: - * In a specific package a set of dependencies need to be associated to a dependency group. - * The semantic of the group needs to be defined (in this case "at-least-one-from-the-group"). +* (D.1) Adding a new tag to the `export` section: e.g. `member_of_group`. +* (D.2) Defining a new tag in the `package` section: e.g. `member_of_group`. -* Use case (2) +#### Example for use case (1) - * A package declares a dependency on a group identifier. - * A set of packages declares that group identifier. +A package declaring a group dependency: -#### Possible Format Extensions +``` + + rmw_implementation + i_am_a_rmw_implementation + +``` -... +A package declaring membership of a group: -## Proposed Solution +``` + + rmw_fastrtps_cpp + i_am_a_rmw_implementation + +``` -### Extensions for the Package Manifest +#### Example for use case (2) -... +A package declaring a group dependency: -### Build Tool +``` + + ros1_bridge + i_am_a_msg_pkg + +``` -For use case (1) the build tool only needs to treat all dependencies of the set as "normal" dependencies. -When they are available within the workspace they should be processed before and being made available to the package declaring the dependency group. +A package declaring membership of a group: -For use case (2) the build tool needs to extract the identifiers from each package and consider them when detemining the topological order. +``` + + std_msgs + + i_am_a_msg_pkg + + +``` -### Packaging process +## Extensions for the Package Manifest -The packaging tool needs to map the dependencies groups and identifiers to the platform specific files as described in the above section. +The modified schema for the manifest will be specified in [REP 149](https://github.com/ros-infrastructure/rep/pull/138). From df8fd2dbe1c9b29f9d3ada146fb3d85589777041 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Tue, 10 Oct 2017 16:45:54 -0700 Subject: [PATCH 3/6] fixup --- articles/102_dependency_groups.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/articles/102_dependency_groups.md b/articles/102_dependency_groups.md index 0b25e284c..ffa18b324 100644 --- a/articles/102_dependency_groups.md +++ b/articles/102_dependency_groups.md @@ -37,14 +37,14 @@ The following subsections will describe two different use cases how dependencies ### (1) Need at least one "Provider" A package might want to declare that at least one dependency from a group needs to be present (called `satisfy-at-least-one`). -Compared to "normal" dependencies it is not requires that all dependencies of that group need to be present to satisfy the dependency group. +Compared to "normal" dependencies it is not required that all dependencies of that group need to be present to satisfy the dependency group. An example for this is the `rmw_implementation` package. -Currently it depends on all RMW implementation packages in order to ensure that it may only be build after all RMW implementation packages have been built. +Currently it depends on all RMW implementation packages (e.g. `rmw_fastrtps_cpp`, `rmw_connext_cpp`, etc.) in order to ensure that it may only be built after all RMW implementation packages have been built. The goal would be that at least one RMW implementation needs to be present in order to build or use the `rmw_implementation` package. -It would be acceptable if more or even all RMW implementations are available but that is not required. +It would be acceptable if more than one or even all RMW implementations are available but that is not required. -In case multiple package declare to be part of that dependency group the `rmw_implementation` package might want to provide an order list of preferred names which should be considered in that order. +In case multiple package declare to be part of that dependency group the `rmw_implementation` package might want to provide an ordered list of preferred names which should be considered in that order. ### (2) All packages of a specific "type" @@ -55,7 +55,7 @@ An example for this is the `ros1_bridge` package. Currently it depends on a manually selected set of message packages. The goal would be that it can declare that it depends on *all* packages which provide messages. -Since the `ros1_bridge` manifest only contains the name of the group dependency in doesn't control which packages declare that they are part of that group. +Since the `ros1_bridge` manifest only contains the name of the group dependency, it doesn't control which packages declare that they are part of that group. It might want to exclude specific packages to be considered even though they are in the group. ## Processes @@ -77,7 +77,7 @@ For the `satisfy-all` semantic it can simply generate dependencies on all packag For the `satisfy-at-least-one` semantic the release tool could either use a platform specific mechanism like `provides` in Debian control files or simply expand the group to all packages (same as `satisfy-all`). This design document doesn't aim to specify what the "best" behavior for the release tool is since the decision is likely platform dependent. -Since the release tool expands the dependency groups when being invoked packages which join the group after that release won't be used until the package defining the group dependency is being re-released. +Since the release tool expands the dependency groups when being invoked, packages which join the group after that release won't be used until the package defining the group dependency is being re-released. ### rosinstall_generator @@ -90,13 +90,13 @@ It could either match the behavior of the build tool or of the release tool or e To provide the necessary metadata for the build and release tools the following information must be provided by the manifest: * (A) A specific package declares a group dependency and uses a name to identify the group. - The dependency can be of any type specified in [REP 140](http://www.ros.org/reps/rep-0140.html): e.g. `build`, `exec`, `test`, `doc`. + The dependency can be of any type specified in [REP 140](http://www.ros.org/reps/rep-0140.html): e.g. `build`, `exec`, `test`, `doc`, etc. * (B) The semantic of the group dependency needs to be defined (in case (1) `satisfy-at-least-one`, in case (2) `satify-all`). * (C) A list of preferred / excluded packages can be enumerated for each dependency group. -* (D) Any package can declare that it is part of that group. +* (D) Any package can declare that it is part of a group by name. ### Possible Format Extensions From 0534486af1398a3410159130183ed4c918dcdbbd Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Wed, 11 Oct 2017 12:54:10 -0700 Subject: [PATCH 4/6] mention buildfarm --- articles/102_dependency_groups.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/articles/102_dependency_groups.md b/articles/102_dependency_groups.md index ffa18b324..98e7666f7 100644 --- a/articles/102_dependency_groups.md +++ b/articles/102_dependency_groups.md @@ -79,6 +79,11 @@ This design document doesn't aim to specify what the "best" behavior for the rel Since the release tool expands the dependency groups when being invoked, packages which join the group after that release won't be used until the package defining the group dependency is being re-released. +### Buildfarm + +Since the buildfarm uses the topological order to trigger downstream binary jobs it needs to consider the dependency groups. +The same interpretation of the `satify` semantic as for `bloom` applies. + ### rosinstall_generator The `rosinstall_generator` already utilized all manifests from the `rosdistro` to decide which packages / repositories should be fetched. From 045cd3f3f3c2435465ec3424f234c5991f750c68 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Wed, 11 Oct 2017 14:39:58 -0700 Subject: [PATCH 5/6] spelling --- articles/102_dependency_groups.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/articles/102_dependency_groups.md b/articles/102_dependency_groups.md index 98e7666f7..6bc7dd4af 100644 --- a/articles/102_dependency_groups.md +++ b/articles/102_dependency_groups.md @@ -23,16 +23,15 @@ Original Author: {{ page.author }} In the ROS ecosystem a ROS package provides information about its dependencies in the manifest. The latest format is specified in the [REP 140](http://www.ros.org/reps/rep-0140.html). -It supports different kind of dependencies (`build`, `exec`, `test`, `doc`, etc.). -But each dependency is mandatory and needs to satisfied on its own. +It supports different kinds of dependencies (`build`, `exec`, `test`, `doc`, etc.) but each dependency is mandatory and needs to satisfied on its own. This article describes the concept of "dependency groups" which could be used for different purposes. In terms of terminology a dependency group is a set of dependencies identified by the name of the group. -For such a group a different semantic can then be applied (instead of just satisfy each dependency on its own). +For such a group a different semantic can then be applied (instead of just satisfying each dependency on its own). ## Use Cases -The following subsections will describe two different use cases how dependencies could be used. +The following subsections will describe how dependencies could be used in two different use cases. ### (1) Need at least one "Provider" From 7d24f9ba37aea3fbbcf64ebabfc40a968e023672 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Thu, 12 Oct 2017 08:03:18 -0700 Subject: [PATCH 6/6] spelling --- articles/102_dependency_groups.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/articles/102_dependency_groups.md b/articles/102_dependency_groups.md index 6bc7dd4af..2dff54996 100644 --- a/articles/102_dependency_groups.md +++ b/articles/102_dependency_groups.md @@ -23,7 +23,7 @@ Original Author: {{ page.author }} In the ROS ecosystem a ROS package provides information about its dependencies in the manifest. The latest format is specified in the [REP 140](http://www.ros.org/reps/rep-0140.html). -It supports different kinds of dependencies (`build`, `exec`, `test`, `doc`, etc.) but each dependency is mandatory and needs to satisfied on its own. +It supports different kinds of dependencies (`build`, `exec`, `test`, `doc`, etc.) but each dependency is mandatory and needs to be satisfied on its own. This article describes the concept of "dependency groups" which could be used for different purposes. In terms of terminology a dependency group is a set of dependencies identified by the name of the group. @@ -59,12 +59,12 @@ It might want to exclude specific packages to be considered even though they are ## Processes -The following two processes need to be update to consider dependency groups. +The following processes need to be updated to consider dependency groups. ### Build tool The build tool needs to consider the declared dependency groups when computing the topological order. -Since it operates on a set packages in a workspace it has all manifests for all of these packages available. +Since it operates on a set of packages in a workspace it has all manifests for all of these packages available. Independent of the semantic of the dependency group (satisfy-all vs. satisfy-at-least-one) the build tool can simply expand a dependency group to all packages of the set. ### Release tool