External protocol specifications with shell based "driver" implementation

[mih]

Capturing an idea that came up in a different context: datalad/datasalad#7 (comment)

Now that we have shell, we could use it to implement handlers for protocols that enable extending and outsourcing functionality via 3rd-party utilities that implement this protocol rather than implementing (Python-based) helpers or subclasses -- exactly what git-annex does for special remotes and backends.

For each such use case we would define a set of operations, and expose them via a protocol. This pattern could be widely applicable. We already archived at this concept (albeit with Python-based implementations) for

• URLOperations
• ArchiveOperations

Other use cases are:

• siblings: we can do create, but nothing else. We need/could have:
  • remove: remove-sibling-* datalad#5946
  • a push/fetch operation could enable the implementation of non-Git remotes
  • more general: API to "configure/enable" our special remotes datalad#3839
• credentials: we have a new approach here, but it is already asking for more formalization RF CredentialManager to have a formal backend class #224
• git-annex special remote driver: git-annex is the only software that implements the git-annex custom remote protocol. Having a second implementation would not be a bad thing. Some special remotes might even be usable for non-git-annex functionality...