Skip to content

Latest commit

 

History

History
977 lines (664 loc) · 31.4 KB

rep-0110.rst

File metadata and controls

977 lines (664 loc) · 31.4 KB

REP: 110 Title: SCM-like rosinstall command structure Author: Thibault Kruse <[email protected]> Status: Draft Type: Informational Content-Type: text/x-rst Created: 11-Jun-2011 Post-History: 11-Jun-2011, 10-Aug-2011, 03-Mar-2012

Table of Contents

  1. Abstract
  2. Specification
  3. Use-cases
  4. Analysis
  5. Rationale
  6. Motivation
  7. Compatibility
  8. Design-Alternatives
  9. Reference-implementation
  10. References
  11. Copyright

Abstract

This REP presents an alternative to the rosinstall [1] command-line interface to a command-based syntax. The command-based syntax is inspired by SCM tools like svn, git, and hg as well as several other ros commands like rosnode or rosservice, and is intended to improve the usability of rosinstall as well as expose more functionality. Common use cases of rosinstall are described and compared with the proposed changes.

Specification

Command-line syntax

The name "rosinstall" does not fit into an SCM like tool as it contains a verb "install" in its name, which is confusing. The name rosws for rosworkspace is a suggestion based also on other activities around rosinstall. (see https://code.ros.org/lurker/message/20110711.160222.666ecfe4.en.html)

The command by default uses the context to determine which folder to use as target of the operations. Two location types are relevant, the workspace folder, and folders of entries.

The workspace folder must contain a .rosinstall file to be recognized as such. The rosws command will consider the first such folder in the chain of the current directory and its parents. The rosinstall command may also rely on the ROS_WORKSPACE environment variable to specify a workspace. If both approaches detect a suitable folder, the command will fail with an error message indicating the ambiguity. The user may specify a target workspace folder with the same option for all commands. This is the preferred approach for scripting with rosws.

To determine what entry a user wants to work on for certain commands, the concept of a localname is used in what follows. The localname is historically an absolute filepath or a relative path relative to the workspace folder. Commands which accept a localname argument will first attempt to resolve that argument against the list of known localnames. If that fails, the commands will fall back to choosing a workspace entry which has the same path in the filesystem, even if the localname does not match. As an example, rosws diff . In a tree with localname foo is equivalent to rosws diff foo. Finally, some commands may fall back to interpret such an argument, if it is the only one, and in the absence of a specifying option, as a workspace folder if the command also accepts a workspace folder.

rosws init [PATH [URI]]

The command creates an initial environment for the other commands to work. This is analogous to scm syntax. Its main purpose is to explain the tool to the user, who understands that the location used here is going to be a special location further on.

The command also generates three initial files, setup.sh, setup.bash and setup.zsh, which when sourced set up environment variables according to the .rosinstall file in the same folder. Those files wll only be overwritten by the rosws regenerate command.

--catkin, --cmake-prefix-path

Options retain semantics as in rosinstall

rosws merge [URI]

This command processes several entries from a given URI source

The command merges new entries (command line or changed in .rosinstall) with existing entries (if a .rosinstall is present). The strategy is to replace existing entries rather than adding whenever possible. This is the case when an existing and a new entry have the same target path, the same SCM type, and a similar SCM configuration (e.g. same URI root).

The strategy fails whenever there are two entries with the same target path but replacing is impossible with the strategy above.

When successfull, the command will append the new entries to the end of the .rosinstall file.

URI can be a path to a rosinstall file, to a local directory containing a .rosinstall.

The command will fail when there are entries with duplicate target paths.

The command calculates the change to the configuration. If the change merely consists of adding entries, this is executed. Else, the user is presented with an overview of planned changes, roughly similar to what the linux aptitude tool may present. The user can then interrupt the operation. An option exists to not ask for confirmation.

Options:

The merge options apply whenever two elements have equivalent localnames (pointing to the same directory)

--merge-keep Existing entries will remain unchanged
--merge-replace
 Existing entries will be overwritten in place
--merge-kill-append
 Exiting entries will be removed, new ones appended to the end
--confirm-all Do not ask for confirmation
-
A single dash can be used for piping stdout into the merge command

rosws set [LOCALNAME] [URI]?

The command serves to add or modify a single entry of the config and is in particular aimed at users entering the entry information themselves rather than relying on some other source of information.

For adding an entry, the common case is to add an SCM entry. The syntax would then be

rosws set localname uri --scmtype --version=version

Options:

--svn Subversion SCM
--hg Mercurial SCM
--git git SCM
--bzr bazaar SCM
--tar tar as source (unofficial experimental feature)
--detached no SCM
--version-new the version specification to use

- single dash means read from stdin (similar to tar command)

Option -version would go against the GNU standard of a global --version option giving the version of a command line tool.

Examples:

$ rosws set robot_model --hg https://kforge.ros.org/robotmodel/robot_model
$ rosws set robot_model --version robot_model-1.7.1
$ rosws set robot_model --detached

rosws update [LOCALNAME]*

The command checks out or updates the given subtrees or all subtrees from their respective SCM provider and URL.

--continue-on-error
 Continue despite checkout errors
--delete-changed-uris
 Delete the local copy of a directory before changing uri.
--abort-changed-uris
 Abort if changed uri detected
--backup-changed-uris=BACKUP_CHANGED
 backup the local copy of a directory before changing uri to this directory.

Options retain their semantics from current rosinstall.

Examples:

$ rosws update -t ~/fuerte
$ rosws update robot_model geometry

rosws info [LOCALNAME]*

This serves to explain the current environment to the novice user, and provide an overview of it. It is supposed to give an overview of the environment and its state. The reference implementation lists the interpreted ROS_ROOT and the ordered paths in the ROS_PACKAGE_PATH. It also provides status information about the local directories, such as whether they exist and whether their current version matches the .rosinstall.

Options:

--data-only just the data, no headers or text
--no-pkg-path does not print ROS_PACKAGE_PATH
--pkg-path-only
 Shows only ROS_PACKAGE_PATH separated by :. Supercedes all other options.
--only Shows only given comma-separated attribute(s) separated by ,
--yaml prints rosinstall format yaml for backing up the current config

These options are mainly intended to allow scripting based on rosinstall.

Examples:

$ rosws info -t ~/ros/fuerte
$ rosws info robot_model
$ rosws info --yaml
$ rosws info --only=path,cur_uri,cur_revision robot_model geometry

rosws remove [LOCALNAME]+

removes entries with given localnames from the .rosinstall file

rosws status [LOCALNAME]*

This command batch-calls SCM status. The output is the concatenation status commands for all versioned entries.

Options:

--untracked also shows untracked files

rosws diff [LOCALNAME]*

This command batch-calls SCM diffs. The output is the concatenation of diff commands for all versioned entries.

Such a feature was suggested on ros-developers. The reference implementation shows how this could be done. However the solution is neither very robust nor flexible for SCM diff options.

There is room for ambiguity when the version specified in the .rosinstall does not match the version in the directory. A warning should be given on stderr at least.

rosws regenerate

This command regenerates the setup.sh, setup.bash and setup.zsh files that setup a shell environment. The rosinstall command overwrites these files on each invocation for historical reasons.

There is as of today no need to regenerate these files even if the .rosinstall file changed in acceptable use-cases. The only cases where the files would be changed is when configuring a workspace against a changed ROS distribution, or after an update of the rosws tool.

Options

--catkin --cmake-prefix-path

Options retain semantics as in rosinstall

Analysis

Use cases

The following are a list of use cases for rosinstall. U1-U6 are based on use cases described in the rosinstall documentation [1].

The new command syntax examples usually have several alternatives to use context or options to specify the workspace to operate on..

U1: Create a new environment based on an existing ros installation

E.g. Developing on top of boxturtle shared install

rosinstall ~/workspace /opt/ros/boxturtle http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall

New command example:

rosws ~/workspace /opt/ros/boxturtle
cd ~/workspace
rosws merge  http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall
rosws update

U2: Create a new environment based on a rosinstall file only

E.g. Full source checkout

rosinstall ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall

New command example:

rosws ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall

U3: Create a new environment with an additional stack

E.g. Developing a stack against a full tree

rosinstall ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall my_stack.rosinstall

New command example:

rosws ~/workspace http://www.ros.org/rosinstalls/boxturtle_pr2all.rosinstall
cd ~/workspace
rosws merge my_stack.rosinstall
rosws update

U4: Adding more entries to an existing workspace

E.g. Adding a rosinstall layout to an existing workspace

rosinstall ~/workspace http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall

New command example:

cd ~/workspace
rosws merge http://www.ros.org/rosinstalls/wg_boxturtle_devel.rosinstall
rosws update

U5: Updating a rosinstall tree

rosinstall ~/workspace

New command example:

cd ~/workspace
rosws update sometree

U6: Changing the version of a specific stack

You can manually edit the file .rosinstall in a rosinstall managed workspace and then call rosinstall ~/workspace.

New command example:

cd ~/workspace
rosws set stackname --version=newversion

U7: Overlaying a released ROS package with the sources

e.g. taking the sources from trunk, using roslocate

roslocate info common_msgs | rosinstall add .

New command example:

cd ~/workspace
roslocate info common_msgs | rosws merge -

U8: Rearranging the order of entries in the ROS_PACKAGE_PATH

You can manually edit the file .rosinstall in a rosinstall managed workspace and then call rosinstall ~/workspace

U9: Removing directories from the ROS_PACKAGE_PATH

You can manually edit the file .rosinstall in a rosinstall managed workspace and then call rosinstall ~/workspace

New command example:

cd ~/workspace
rosws remove foo

U10: Checking the validity of the current ROS environment

using diverse file exploration tools

U11: Creating a snapshot of the configuration

rosinstall ~/workspace --generate-versioned-rosinstall=GENERATE_VERSIONED

New command example:

cd ~/workspace
rosws info --yaml > GENERATE_VERSIONED

U12: Modifying the current configuration according to a snapshot

same as U4

U13: Basing the environment on a different ROS_ROOT

Users should generall avoid doing this, as it can lead to failure in many ways.

Therefore not implementing the usecase in this REP, meaning user has to manually modify .rosinstall and call rosinstall

U14: Create a new environment based on an existing one

rosinstall ~/workspace /path/to/other/workspace

New command example:

rosws init ~/workspace /path/to/other/workspace

U15: Moving a local stack to a different location

The user has some ROS stack in directory foo, and want to move that stack to directory bar. This means moving the files as well as updating the .rosinstall and the setup.sh.

Will not be supported for now.

U16: User makes arbitrary changes to .rosinstall

An expert user can change path, version, scm, entry order, etc. in his rosinstall. In some cases he then just wants his setup.sh file to be regenerated, in other cases he may want to update directories to specific revisions.

rosinstall ~/workspace

New command example:

cd ~/workspace
rosws update foostack
rosws regenerate

U17: User wants to see combined diff

When the user works on several interdependent stacks, he wants to create a unified diff of those stacks against their checked-out revision.

rosinstall ~/workspace --diff

New command example:

cd ~/workspace
rosws diff

U18: User wants to status of scm files

When the user works on several interdependent stacks, he wants to see what files are currently in a changed status in their respective SCM.

rosinstall ~/workspace --status

New command example:

cd ~/workspace
rosws status

Use-case analysis

All rosinstall use-cases deal with a user trying to maintain a ROS environment on a local machine. A ROS environment means the assignment of values to ROS_PACKAGE_PATH, PATH, PYTHONPATH and ROS_MASTER_URI, as well as further changes to support ROS toolchains.

Given the description, a user of rosinstall needs to be aware of the following artifacts:

  • any .rosinstall file in the current directory
  • the setup.* files in the current directory
  • the state of local directories
  • the information contained in other rosinstall files

As a user is free to manipulate entry in the configuration files, a major cause for errors are mistakes during such modifications. As an automated tool, rosinstall currently only knows of this: The user may have changed the rosinstall file, the user may have provided additional locations, now a new .rosinstall file may have to be merged with additional locations, and new setup.* files have to be generated.

An early attempt for this REP tried to provide different commands for different use-cases, such as init, remove, add and update. However the code for all those commands had in all cases to remain largely similar to cope with the situation where a user had manually changed his or her .rosinstall, or filesystem.

The list of use-cases thus only displays the range of distinctive usages of rosinstall, but does not provide help into splitting up the main functionality into many small commands.

T1: Table of use cases and commands

UC init merge set info update diff status remove
U1 x x     x      
U2 x              
U3 x x     x      
U4   x     x      
U5         x      
U6     x   x      
U7   x     x      
U8                
U9               x
U10                
U11       x        
U12   x     x      
U14   x     x      
U15                
U16         x      
U17           x    
U18             x  

Motivation

The rosinstall tool, while useful, has a command-line syntax that can be difficult for beginners of ROS to become familiar with. Users who are intimidated by rosinstall can resort to manually creating source trees and maintaining ROS_PACKAGE_PATH by hand. Improving the usability of the rosinstall tool can improve the ROS experience for beginner users as well as introduce advanced users to more powerful features.

Rationale

The extended command structure allows to perform a broader range of operations concerning multiple SCMs, and mere investigative commands.

Breaking down functionality helps self-explaining of the tool.

Also this makes it easier to extend rosinstall with other functions, and to use rosintall in scripts (e.g. provide a curses/tk-based rosinstall ui).

Another problem are the --options. Those are only required in specific use-cases, but in the current solution the user may provide them in any case, and thus the user will have difficulties to tell what his choices for options are, given his intention. Having distinct commands allows a help function to quickly tell the user what options are available for his specific intention.

The SCM like structure also allows code completion to provide command choices, which is a significant help for users when learning a new tool.

The vision of the source code is that it will one day in large parts be moved out of rosinstall to provide ROS-independent functionality for managing directories of mutliple SCMs. However the rosinstall functionality of generating setup files for ROS environments will remain important. Therefore a name change to rosws is suggested rather than a ROS-independent name.

Names

The current rosinstall command structure allows a command such as:

rosinstall ~/workspace /opt/ros/diamondback ~/workspace2 ~/workspace/foo ~/download/temp.rosinstall

The intention of the user typing this cannot be inferred from the command, nor can what the tool will do be inferred from the syntax. The key idea is to have a syntax that allows the user to express a specific intention and to understand what the tool will do.

The commands info, diff, and status are self-explanatory.

The word install was dropped completely with the broad functionality it offers in rosinstall. install makes users shy away from usage fearing they might corrupt their debian installation with a tool.

The set and merge commands are somewhat ambiguous, as the user would need to use them for multiple purposes of adding entries or modifying versions of entries. However the different use-cases are algorithmically so similar that own commands for each seem superflous.

Rejected names include:

add, edit, modify, read, load, import, change

Backwards Compatibility

The rosinstall setup files remain unchanged.

Current rosinstall syntax is:

Usage: rosinstall PATH [<options> ...] [URI]...

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -n, --nobuild         skip the build step for the ROS stack
  --rosdep-yes          Pass through --rosdep-yes to rosmake
  --continue-on-error   Continue despite checkout errors
  --delete-changed-uris
                        Delete the local copy of a directory before changing
                        uri.
  --abort-changed-uris  Abort if changed uri detected
  --backup-changed-uris=BACKUP_CHANGED
                        backup the local copy of a directory before changing
                        uri to this directory.
  --generate-versioned-rosinstall=GENERATE_VERSIONED
                        generate a versioned rosintall file

--generate-versioned-rosinstall is a special case and is replaced by rosinstall snapshot.

This will be fulfilled by rosws install.

The change in command syntax impacts rosinstall based tools such as the new proposal for a rosworkspace.

Design Alternatives

rosinstall0.5

As originally developed by Tully Foote.

Commands are:

rosinstall PATH [options] [URIs]

with a 4 Phase model, Merging, Checking out, Generating setup, bootstrapping ROS serving most use-cases and --generate-versioned-rosinstall being an exceptional command

Extended-options

Similar to the commands provided in the REP, some of the functionality could be provided by additional --options.

e.g.:

rosinstall --info
rosinstall --check
rosinstall --diff
rosinstall --status

This is less useful for bash completion and focused help.

SCM-like-explicit-path

As initially suggested for this REP, all commands require the path on which to work as the first argument.

Commands are:

rosinstall <command> PATH [options]
rosinstall <command> PATH [options] [URIs]

The SCM-like Designs are in line with the other ROS tools. However, SCM tools usually drop the PATH argument and work in the current directory tree.

SCM-like-implicit-path

Inferring the path from context can be done to varying degrees, like ignoring the ROS_WORKSPACE variable or only considering the current folder, not its ancestors as candidates.

Another alternative is to try first interpreting any first argument as a workspace folder, however this may conflict with the syntax of command set.

SCM-like-extended-command-set

The install command suggested provides support for several use cases which could be further split up into different commands, e.g. add and apply. See Rejects.

Reference implementation

Reference implementation is provided with the latest rosinstall release.

Rejects

Ideas for rosinstall commands that were dropped for this REP, but may become interesting later:

2nd draft rejects

In 2012 there was a second surge to realize the REP. In this it was decided to make a more significant step away from the rosinstall design. As a consequence, certain ideas were brought up and rejected:

  • rosws install: replicates rosinstall

A command doing many things at once confuses and scares the occasional user Similarly commands set, merge nad update only do one thing, and do not allow doing more than one thing.

  • update after set/merge

It could be convenient if the tool did or offered to checkout/update tree entries after a set/merge operation. This can be added as an option in the future if users want it.

  • many uris for init and merge

These commands could technically take many uri arguments instead of maximally one. The decision to allow only one is to help users understand the syntax. The problem with many arguments we saw with rosinstall is that users could not read from the syntax whether the second argument had a special semantic or not, or whether the order of the arguments mattered.

  • Maintain rosinstall assumptions about ROS

rosinstall was provided in support of early ROS releases, this REP is provided in the wake of ROS fuerte, which makes significant changes to how a ROS environment is set up. rosws can have a fresh start with fuerte only providing minimal support for ealier distros. In particular rosws will not insist on having a ROS stack, and will not build ros or ros_comm even after source checkouts, nor have an option to do so. A future development could be to even drop generation of the setup.* files.

  • optional localname when adding using set

With SCM providers, it is possible to checkout a new local copy from a url without giving a target folder name. The same could be achieved with the rosws set syntax. However this makes the syntax more confusing, and in many cases we want the user to be aware of names because the repository names often have dashes where the localname should have underscores.

  • rosws snapthos

The command is just a variant of rosws info and can be a --option

  • making set and merge a single command

Both commands add/modify entries, so the case was made to unify them. It was rejected after lengthy discussions for these reasons: There is the fact that the 1st argument is mandatory for set, but not useful as localname for merge. There are the different -- options for the different cases. For the user, it should be clear whether his focus is some "remote" resource containing a set of many yaml entries, or on a single entry to add/change. That split is also the split of commands. The user won't be surprised by tab completion options --hg (or just hg as positional argument) or --version when merging with rosinstall files. They would make no sense, but code completion cannot know that. The localnames wont get in the way of tab completion user when wanting to point to a local xyz.rosinstall file. It is true that merge is semantically like n calls to set, except it is an atomic operation (all succeed, or all fail). In the far future, we might think of nested workspaces having state and version of their own and defining commit() and add()and update() actions. Then we could have a different type for the "set" command, BUT this would be very different semantics from the merge command, and the merge command would not become obsolete. This possibility should also indicate why merging the set and merge command above would hinder things in the future.

  • making all entry arguments in set mandatory

The set semantics could be cleaner if it followed the principle that always a full entry (scmtype, uri, version) must be provided. This is however unnecessarily tedious when the user just wants to change one of the details of an existing entry

  • splitting set into add and edit

The split has the advantage that more warnings can be provided in case the user tries to add an entry which overlaps another. However a confirmation step is sufficient. Also this would allow having mandatory positional parameters for scmtype and uri in the add case, and optional ones for the edit case, but two different syntax structures for very similar cases would be confusing as well.

  • making the scmtype in set a positional argument

positional arguments like hg, git and svn do not work well with an optional third positional argument URI.

  • detach as a command of its own

set has option --detached to remove all scm information. The case was made to make this a command of its own, in particular for a fuse set/merge command. But it fits better with the set - merge split.

  • remove as a set option

removal is an instance of modifying, so it could be unified with set as a --option. However the end effect of a set operation is always that something exists afterwards, so these semantics get muddied by remove as an option.

  • default merge option killAppend

When using merge or set, a decision is when an overlapping entry exist, whether to replace it in place, or whether to remove the existing and append the new one at the end. rosinstall has the latter semantic because it conveniently implies that it is possible to ensure an ordering of entries in the config file by ordering them in the argument list.

For novice users however this behavior is confusing and also hard to describe. The default semantics for merge is therefore to replace the entry in place. An option exists for merge to change that behavior.

1st draft rejects

The following attempts were rejected in the first REP draft in 2011

  • rosws regenerate: Generates setup files but does not attempt to update SCM based entries
  • rosws init: first time command demands a ROS root to be specified. too similar to install
  • rosws update: merges new entries and runs update on all scm based entries. too similar to install
  • rosws remove: removes an entry from .rosinstall. Better be done by hand

In the second draft, those command were resurrected in particular in the absence of an all encompassing rosws install command.

Those commands were too similar to rosws install and their behavior to bulk update and regenerate all trees conflicted with the semantics sugested by their name.

  • rosws overlay: looks up repository location based on roslocate and rosinstalls it
  • rosws change-ros: allows to move from e.g. /opt/ros/diamondback to ~/ros or /opt/ros/cturtle
  • rosws mv: Moves local checked out stack to different location on filesystem
  • rosws reorder: changes the order of stacks in ROS_PACKAGE_PATH
  • rosws change-version: changes the desired version of an SCM based local stack
  • rosws check --repair: Attempts to (interactively) fix inconsistencies such as duplicate paths

Those commands make the initial version of the REP more complex to discuss and harder to implement, therefore they were left out.

References

[1](1, 2) rosinstall (http://www.ros.org/wiki/rosinstall)

Copyright

Copyright (c) 2011 by Thibault Kruse. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v0.1 or later (the latest version is presently available at http://www.opencontent.org/openpub/).