kci command line syntax

[gctucker]

The new kci command line tool was initially put togher as a quick way to bootstrap development of the new API & Pipeline without relying on raw URLs and hacky scripts. As part of preparing a first release candidate for the new API wich includes the kernelci Python package, we should revisit its syntax and list the features we want it to provide.

As identified with the 2020 Community Survey, 70% of users want to have a web dashboard as part of their daily workflow. This basically means that about a third of the users would rather just use a command line tool or an API library. It's also extremely useful to have such tools when developing test workloads before automating them with deployed services. So it is a key part of the whole KernelCI user experience.

Moving away from the legacy kci_build, kci_test etc. collection of tools to a single entry point kci is already a clear improvement. It does however add one layer of subcommands, which argparse wasn't really designed to deal with. This is where the #258 discussion comes from. In this new discussion, we'll try to ignore the implications of any particular framework initially and just focus on the syntax purely from a user point of view.

Roadmap issue: kernelci/kernelci-core#2125

Note: The TOML settings file can provide default values for arguments that aren't specified directly on the command line. This is described in more detail as part of #261.

[gctucker]

Common principles

kci top-level arguments

Some arguments can be defined globally for the overall kci tool rather than particular subcommands. For example, specifying the location of the TOML settings file:

kci --toml-settings=PATH [SUBCOMMANDS]

This optional as it has default locations defined in the code.

Positional arguments

Generally speaking, positional arguments may be used when an obvious value needs to be provided and it wouldn't make sense to store it as part of the TOML settings. For example, when retrieving a single node with kci node get ID the node ID is required and specific to the command. Having kci node get --id=ID would just make the syntax more complicated and longer with no added benefit.

Arguments that are required but have a good use-case for being in the TOML settings are better defined as named arguments. For example, the API configuration is required when running any command involving the API. Providing it all the time would be cumbersome, so having --api makes more sense. It can then be saved as kci.api in TOML to provide a user-defined default value.

Paginated results

Some commands can return a lot of data, for example when looking for nodes. The API provides pagination for these use-cases, with a starting offset and a number of items to include. The API terms are offset and limit. From a user experience point of view, these may not be the most obvious terms. Could limit be the index of the last item, and could offset be the index difference with limit? It would seem worth solving this ambiguity and also having short options such as -x and -y as these are likely to be regularly used. Also, when the output is a TTY, the command could dynamically use paging if the user uses page up/down keys to scroll through the entries to avoid the command line syntax issues entirely.

Initial proposal: --offset, --limit
Alternatives:

• --seek, --length
• -n for number of objects, -p for page number e.g. -n10 -p3 means indices 30 to 39 (page number starting at 1)
• -l for page length, -n for page number (and --page-length, --page-number for long options)

[gctucker]

About this part:

kci --settings=PATH [SUBCOMMANDS]
kci --pipeline=PATH [SUBCOMMANDS]

I'm not sure --settings and --pipeline are good argument names. The --pipeline argument doesn't obviously mean it's the YAML configuration, and if we call it --config then it's confusing with also --settings.

One way to remove this ambiguity is to have longer argument names e.g. --user-settings and --yaml-config, maybe with some short options as well to make them easier to use without typing long arguments all the time. Another way to improve this is to rely on the help message, but that's probably not as good as having clear argument names.

[gctucker]

Gathering here some common one-letter arguments from various classic tools:

• -h: help (built-in anyway)
• -v: verbose
• -q: quiet
• -s: silent
• -i: interactive, include
• -r: recursive, reverse
• -f: force, file
• -a: all, append
• -y: yes (non-interactive)

[gctucker]

So for the issue of settings and config, I would propose this:

• --toml-settings, -s for the TOML user settings
• --yaml-config, -c for the YAML pipeline configuration

Both TOML and YAML are well established standards so they should keep being supported for a very long time. Then if we wanted to switch to a different format, having the format name in the long option actually helps with this e.g. we could have --xml-config if the pipeline configuration was stored in XML.

I'll be using this in the implementation PR for now, but please put any comment here if you think another approach would be better.

[gctucker]

For paginated results, in fact we might just be able to rely on click.echo_via_pager():
https://click.palletsprojects.com/en/8.1.x/api/#click.echo_via_pager

I've used it in the new kci config implementation: kernelci/kernelci-core#2138

[gctucker]

The echo_via_pager() solution only works for showing data that has already been sent by the API, obviously. So it doesn't really solve the issue of getting paginated data from the API.

I've now implemented the 3rd alternative with -n for page number and -l for page length, with corresponding long options --page-number and --page-length:
kernelci/kernelci-core#2178

[gctucker]

Requirements

Usability

A number of usability requirements apply to the tool in general through all its features:

• built-in documentation: The tool should provide help and documentation by itself.
• lightweight dependencies: The tool should be easy to install on major distros e.g Debian Stable.
• offline mode: It should be possible to run jobs without a live API, using local files only.

Features

The kci command line tool needs to provide at least support for these main features:

• user accounts: manage profile, password
• user groups: create, delete groups and add, remove users in groups
• config: browse, edit, validate the YAML configuration
• jobs: generate and schedule jobs
• nodes: find, edit, submit nodes
• events: subscribe, receive and publish events
• storage: upload, download, list files in storage

Each feature is fully described with an individual section in this discussion. Additional high-level features may also be implemented as a follow-up since these aren't strictly needed for the first production release candidate:

• data visualisation: showing data in a human-friendly way and interactively
• bisection: performing semi-automated Git bisections
• issue tracking: updating issues related to kernel regressions
• email reports: generating and sending customised email reports

[gctucker]

Since we've lost the kci api hello command, the Early Access documentation page is kind of broken. It would probably still make sense to have a kci api sub-command to get things such as the API version info, the URL and other details like this. So I'll prepare a PR for this with just kci api hello for now.

[gctucker]

Update: This has now been fixed and the documentation has been updated accordingly. For example:

$ ./kci api hello --api=staging
{
  "message": "KernelCI API"
}

[gctucker]

User accounts

Each user of the API has their own dedicated account. As such, the tool needs to provide ways to perform all the related house-keeping tasks: generating API tokens, updating profile details, changing passowrds, joining / leaving groups (with admin rights or pending admin moderation).

Command	Description	Scope
kci user token USERNAME --api=API	Create an API token
kci user password update USERNAME --api=API	Update the user password interactively
kci user password reset USERNAME --api=API	Request a password reset over email
kci user find KEY=VALUE KEY=VALUE... --api=API	Find users matching search attributes
kci user whoami --api=API	Show the current user's details
kci user verify --api=API	Verify the user's email interactively
kci user update FIELDS --api=API	Update the current user profile
kci user update FIELDS --api=API --username=USERNAME	Update any user profile	admin
kci user add USERNAME --api=API	Add user account interactively	admin
kci user deactivate USERNAME --api=API	Deactivate a user account	admin
kci user activate USERNAME --api=API	Activate a user account	admin

[gctucker]

It seems like adding the group commands would make sense here. Then we would have:

Command	Description	Scope
kci user group find KEY=VALUE... --api=API	Find groups matching search attributes
kci user group leave NAME --api=API	Leave a group
kci user group join NAME --api=API --user=USERNAME	Add a user to a group	admin
kci user group add NAME --api=API	Create a new group	admin
kci user group delete NAME --api=API	Delete a group	admin

Alternatively, joining and leaving groups may be done with kci user update but this requires providing the full list of groups with the group=group1 group=group2... syntax which isn't ideal. So having a dedicated command might be better.

[JenySadadia]

The proposal for kci user group looks good to me.

[JenySadadia]

As I started implementing these commands, I think we should eliminate kci user group show command as we can get details of a particular group by kci user group find name=<group-name> command as well.

[gctucker]

Yes, and if we wanted such a feature it should be called kci user group get instead to be consistent. I'll remove the row from the comment above.

[JenySadadia]

Thanks.

[gctucker]

User groups

Users can belong to groups in order to share permissions for updating node data. The permissions scheme is not entirely finalised yet, only the main features are well understood. Users can be admins in which case they can generate an API token with admin scope.

Update: This has now been consolidated with kci user as per #263 (reply in thread)

Command	Description	Scope
kci user group find KEY=VALUE... --api=API	Find groups matching search attributes
kci user group leave NAME --api=API	Leave a group
kci user group join NAME --api=API --user=USERNAME	Add a user to a group	admin
kci user group add NAME --api=API	Create a new group	admin
kci user group delete NAME --api=API	Delete a group	admin

[JenySadadia]

kci group join NAME --api=API --user=USERNAME --scopes=SCOPES

What is the significance of scopes here?
Bdw adding a user to a group can be done with kci user update command too.

[gctucker]

Scopes are the same as in the JWT. Right now, we only have admin afaik.

But yes I'm not sure if we can have scopes for a particular group only. So maybe it's better to remove that from the kci group join command for now and just have global scopes initially. I was thinking, it would be good to have users with permission to add other users to a particular group but not any group. But that will need some clarification, and it should be a backwards-compatible additional feature we can work on later on.

[gctucker]

Note: I've updated the proposal without kci group join NAME --scopes=SCOPES as per the previous comments.

[gctucker]

See my comment in the kci user section, I think it would be good to have groups as a subset of kci user to make it clear this is about user groups.

[gctucker]

Jobs

A "job" is anything that runs, performs some actual task which generates some data and submits it to the API. Typical tasks performed by jobs are building kernels, running tests and post-processing existing data from other jobs.

The NODE and PARENT arguments can be either node IDs or paths to local JSON files when working "offline" without a live API instance. The schedule command goes through all the steps to get a job to run in one go. The new, generate and submit commands break down this process into individual steps which is mostly helpful during development and when debugging issues.

The RUNTIME argument is a string designating either a runtime by its name e.g. k8s-gke-eu-west4 or a type of runtime e.g. lava.

Command	Description
kci job schedule NAME PARENT RUNTIME --api=API --platform=PLATFORM	Schedule a job (create node, generate and submit)
kci job new NAME PARENT RUNTIME --api=API --platform=PLATFORM	Create a new job node
kci job generate NODE --api=API --output=OUTPUT	Generate a job definition in a file
kci job submit DEFINITION	Submit a job definition to its designated runtime
kci job wait NODE	Wait for the job to complete

[gctucker]

Config

The KernelCI YAML configuration is used to define what an automated pipeline should do. Some base YAML files are provided alongside with the Python package to define common things such as Linux kernel upstream Git repositories and standard API instances. This config data is also required when using the command line tools by hand, for example to load the API instance URL and version, storage and runtime details etc.

Command	Description
kci config validate	Run validation checks over the whole YAML configuration
kci config dump PATH --depth=1	Show the data loaded from the YAML configuration found at the end of a provided path
kci config set PATH=VALUE	Set a value in the YAML config at the provided path

In this context, PATH can be a dotted string value with the hierarchy to reach an entry. This works well for dictionaries, for example: to show the URL of the early-access API config: kci config dump api.early-access.url. For lists we could have an index e.g. scheduler[0].job.

[gctucker]

Nodes

Command	Description
kci node find KEY=VALUE... --offset=OFFSET --limit=LIMIT --api=API	Find nodes with attributes matching the key/value pairs
kci node count KEY=VALUE... --api=API	Count the number of nodes matching the key/value pairs
kci node get ID --api=API	Get a node with the given ID
kci node submit --api=API	Submit a new node or update an existing one, reading the data from stdin
kci node edit ID KEY=VALUE... --api=API	Edit a node matching the given ID with the provided key/value pairs or interactively

Commands such as kci node find could be showing one line per node found with the ID, name and other attributes not listed in the key/value pairs to help the user find a particular one. We could also have a --json boolean flag to print the full JSON raw data on stdout for other use-cases, or maybe a --format switch to choose between JSON, a plain list of IDs with one per line or a human-readable form.

[JenySadadia]

kci node submit and kci node update should also have the option to use json file input.

[gctucker]

I think submit always has to take JSON from stdin, and update is only for specifying fields on the command line. If you wanted to use JSON to change some fields, you could do a get, edit the data and then submit. If you wanted to update some fields in one go, you could just use update. Maybe kci node edit would be a better name, it could also provide an interactive mode to edit the fields when no key/value pairs are provided.

[gctucker]

Note: I've replaced kci node update with kci node edit and mentioned potentially having an interactive mode.

[gctucker]

Events

There are two main event-driven use-cases: broadcast publisher/subscriber event channels for general purpose communication and unicast event queues which are typically used for load balancing. The same events can be sent to both types of subscribers. The majority of them gets generated automatically by the API whenever a node gets created or updated. The name of the channel matches the type of data for API-generated events: node, user etc. The list of available channels can be retrieved with kci event status.

Queues are effectively a way to group event listeners which all share a common purpose. Only one random listener in that group will receive each message. Queues have arbitrary names and belong to one by design to avoid collisions between users (this may be revisited in the future if load balancing needs to happen across multiple users). Queues need to be opened and closed by the user. Then listeners don't need to subscribe to queues, they can just push or pop events as they're guaranteed to be delivered unlike pub/sub ones. A typical use-case is load balancing, for example when running multiple identical scheduler services but only of them should process the event to schedule jobs.

Command	Description
kci event join CHANNEL --api=API	Subscribe to a broadcast pub/sub channel
kci event leave CHANNEL --api=API	Unsubscribe from a broadcast pub/sub channel
kci event send CHANNEL --api=API	Send an event to a broadcast channel from stdin
kci event receive CHANNEL --api=API	Wait and receive an event from a broadcast channel
kci event filter CHANNEL FILTER=VALUE... --api=API	Set up filters for a broadcast pub/subchannel
kci event open CHANNEL QUEUE --api=API	Open a unicast queue
kci event close CHANNEL QUEUE --api=API	Close a unicast queue
kci event push CHANNEL QUEUE --api=API	Push an event to a unicast queue from stdin
kci event pop CHANNEL QUEUE --api=API	Wait and pop an event from a unicast queue
kci event status --api=API	Show the current subscriptions with filters etc.

Here's an example for pub/sub:

kci event join node
kci event filter node state=done
kci event receive node
# ... wait and receive an event for a node that changed to state=done
cat node-event.json | kci pubsub send node
kci event leave node

And one for an event queue:

kci event open node scheduler-exp
# The next line could be run several times in parallel:
kci event pop node scheduler-exp
# ... wait and receive an event
kci event close node scheduler-exp

To check the status of things:

kci event status
channel  queue          filters
-------  -------------  ----------
node                    state=done
node     scheduler-exp

An alternative would be to have separate commands e.g. kci pubsub and kci queue, but these two things are quite closely related as the events generated by the API are sent both to pub/sub channels and queues.

[JenySadadia]

For unicast messages, as per the current design, a user can use the same /subscribe and /unsubscribe endpoints by sending a list/queue name instead of a channel name.
This would eliminate the requirements of kci event open and kci event close commands.
Also, we can rename kci event join and kci event leave to kci event subscribe and kci event unsubscribe respectively as in both the use-cases(unicast and broadcast) we will be using the same endpoints for subscribing and unsubscribing.

[JenySadadia]

Please see kernelci/kernelci-core#2203 for the rework of existing commands.

[gctucker]

No I really think we need different commands, first I thought we could use the same but came up with the different ones for a reason. I think there should be different API endpoints too. I can explain what I mean in more detail in an issue if you want.

[JenySadadia]

I think we can discuss it further once I create a draft PR with Redis List implementation.
I have introduced /push and /pop endpoints in that one. But used the same endpoints for subscribing and unsubscribing the lists.

[JenySadadia]

@gctucker Could you please take a look at an initial draft for unicast list implementation? kernelci/kernelci-api#417

[gctucker]

Storage

Storage services are hosted separately from the API. They can be of any kind as long as an implementation is available in kernelci.storage. The key requirement is to have publicly-available HTTP URLs for downloading files as the API data schema just needs these URLs for all the artifacts.

Command	Description
kci storage upload FILE [PATH] --storage=STORAGE	Upload a file to storage with an optional destination path
kci storage list [PATH] --storage=STORAGE	List files and directories on storage, optionally within a particular path
kci storage download PATH --storage=STORAGE	Download a file from storage
kci storage url PATH --storage=STORAGE	Get the public HTTP URL of a file from storage

[gctucker]

As per the Requirements section, a number of things will need to be done as follow-ups:

• data visualisation: showing data in a human-friendly way and interactively
• bisection: performing semi-automated Git bisections
• issue tracking: updating issues related to kernel regressions
• email reports: generating and sending customised email reports

However, the initial goal of establishing the basis for the kci command line tool syntax has now been reached so this can be closed. The next step is to complete the implementation as per kernelci/kernelci-core#2125 and update the documentation accordingly.