From 857f815a1b708040b25ae11a2c18b1c961106de1 Mon Sep 17 00:00:00 2001 From: cbruno10 Date: Thu, 6 Feb 2025 16:14:23 -0500 Subject: [PATCH] Add plugin release checklist and fix errors in plugin management doc (#45) --- docs/manage/plugin.md | 27 +-- docs/reference/develop/index.md | 6 +- .../develop/plugin-release-checklist.md | 224 ++++++++++++++++++ docs/sidebar.json | 10 +- 4 files changed, 241 insertions(+), 26 deletions(-) create mode 100644 docs/reference/develop/plugin-release-checklist.md diff --git a/docs/manage/plugin.md b/docs/manage/plugin.md index 5200592..dea939d 100644 --- a/docs/manage/plugin.md +++ b/docs/manage/plugin.md @@ -26,7 +26,7 @@ For example, to install the 0.118.0 version of the aws plugin: $ tailpipe plugin install aws@0.118.0 ``` -This will download the aws plugin version 0.118.0 (the one with the `0.118.0` tag) from the Hub registry. +This will download the aws plugin version 0.118.0 (the one with the `0.118.0` tag) from the Hub registry. ## Installing from a SemVer Constraint @@ -34,7 +34,7 @@ Plugins should follow [semantic versioning](https://semver.org/) guidelines, and The intent of the version tag is that it is immutable - while it is technically possible to move the version tag to a different image version, this should not be done. -Installing with a semver constraint allows you to "lock" (or pin) to a specific set of releases which match the contraints. +Installing with a semver constraint allows you to "lock" (or pin) to a specific set of releases which match the contraints. If you install via `tailpipe plugin install aws@^1`, for example, `tailpipe plugin update` (and auto-updates) will only update to versions greater than `1.0.0` but less than `2.0.0`. @@ -72,7 +72,7 @@ $ tailpipe plugin install aws@2.x.x ``` - To install the latest version locked to a specific minor version: -```bash +```bash $ tailpipe plugin install aws@~2.1 # or $ tailpipe plugin install aws@2.1.x @@ -86,23 +86,6 @@ Tailpipe plugins are packaged in OCI format and can be hosted and installed from $ tailpipe plugin install us-docker.pkg.dev/myproject/myrepo/myplugin@mytag ``` -## Installing from a File - -A plugin binary can be installed manually, and this is often convenient when developing the plugin. Tailpipe will attempt to load any plugin that is referred to in a `connection` configuration: -- The plugin binary file must have a `.plugin` extension -- The plugin binary must reside in a subdirectory of the `~/.tailpipe/plugins/` directory and must be the ONLY `.plugin` file in that subdirectory -- The `connection` must specify the path (relative to `~/.tailpipe/plugins/`) to the plugin in the `plugin` argument - -For example, consider a `myplugin` plugin that you have developed. To install it: -- Create a subdirectory `.tailpipe/plugins/local/myplugin` -- Name your plugin binary `myplugin.plugin`, and copy it to `.tailpipe/plugins/local/myplugin/myplugin.plugin` -- Create a `~/.tailpipe/config/myplugin.tpc` config file containing a connection definition that points to your plugin: - ```hcl - connection "myplugin" { - plugin = "local/myplugin" - } - ``` - ## Viewing Installed Plugins You can list the installed plugins with the `tailpipe plugin list` command: @@ -120,7 +103,7 @@ To update a plugin to the latest version for a given stream, you can use the `t ``` tailpipe plugin update plugin_name[@stream] -``` +``` The syntax and semantics are identical to the install command - `tailpipe plugin update aws` will get the latest aws plugin, `tailpipe plugin update aws@1` will get the latest in the 1.x major stream, etc. @@ -136,7 +119,7 @@ You can uninstall a plugin with the `tailpipe plugin uninstall` command: ``` tailpipe plugin uninstall [plugin] -``` +``` ## Tailpipe Plugin Registry Support Lifecycle diff --git a/docs/reference/develop/index.md b/docs/reference/develop/index.md index c0cf5bb..6ed7ccf 100644 --- a/docs/reference/develop/index.md +++ b/docs/reference/develop/index.md @@ -1,5 +1,7 @@ --- -title: Developers +title: Developers --- -Learn how to ... \ No newline at end of file +# Developers + +- **[Plugin release checklist →](/docs/reference/develop/plugin-release-checklist)** diff --git a/docs/reference/develop/plugin-release-checklist.md b/docs/reference/develop/plugin-release-checklist.md new file mode 100644 index 0000000..7227d99 --- /dev/null +++ b/docs/reference/develop/plugin-release-checklist.md @@ -0,0 +1,224 @@ +--- +title: Plugin Release Checklist +sidebar_label: Plugin Release Checklist +--- + +# Plugin Release Checklist + +All plugins are currently hosted on the [Tailpipe Hub](https://hub.tailpipe.io). If you want to contribute one -- and we hope you do! -- here are the most common things we ask contributors to check to prepare for the plugin's release. Feel free to tick the boxes as you go through the list! + +- [Basic Configuration](#basic-configuration) +- [Configuration File](#configuration-file) +- [Credentials](#credentials) +- [Table and Column Names](#table-and-column-names) +- [Table and Column Descriptions](#table-and-column-descriptions) +- [Table and Column Design](#table-and-column-design) +- [Documentation](#documentation) +- [Final Review](#final-review) + +## Basic Configuration + + Repository name + +The repository name should use the format `tailpipe-plugin-`, e.g., `tailpipe-plugin-aws`, `tailpipe-plugin-azure`, `tailpipe-plugin-github`. The plugin name should be one word, so there are always 3 parts in the repository name. + + Repository topics + +To help with discoverability in GitHub, the repository topics should include: +- duckdb +- sql +- tailpipe +- tailpipe-plugin + + Repository website + +The repository website/homepage should link to the Hub site. The URL is composed of the GitHub organization and plugin name, for instance: +- https://github.com/turbot/tailpipe-plugin-aws results in https://hub.tailpipe.io/plugins/turbot/aws + + Go version + +The Go version in `go.mod` and any workflows is 1.23. + + .goreleaser.yml + +The `.goreleaser.yml` file uses the standard format, e.g., [AWS plugin .goreleaser.yml](https://github.com/turbot/tailpipe-plugin-aws/blob/main/.goreleaser.yml). + + CHANGELOG + +A `CHANGELOG.md` is included and contains release notes for the upcoming version (typically v0.1.0). + + License + +The plugin uses 2 licenses: +- Most of the plugin uses the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) +- Except for the docs which use the [CC BY-NC-ND License 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/deed.en) + + Makefile + +The `Makefile` file is present and builds to the correct plugin path. + +## Credentials + + Terraform compatibility + +If there's a Terraform provider for your API, the plugin supports the same credential methods as the provider. + + Existing CLI credentials + +When there are commonly used CLI credentials, like `.aws/credentials`, the plugin works with them. + + Expiry + +When credentials expire, and the API's SDK does not automatically refresh them, the plugin alerts the user and tells them how to refresh. + + Environment variables + +It's possible to set credentials using an environment variable if the API's SDK also supports using environment variables. + +## Table and Column Names + + Table names + +Table names use the format `_`, e.g., `aws_cloudrail_log`, `aws_s3_server_access_log`, `azure_activity_log`, `github_audit_log`. + + Column names + +Column names should use the lower snake case form of the log property. For instance, in `aws_cloudtrail_log`, the `userIdentity` property becomes the `user_identity` column. + +## Table and Column Descriptions + + Descriptions + +Every table and column has a description. These are consistent across tables. The default `tp_*` column descriptions should be overridden if there's additional log specific information in the respective `tp_*` column. + +## Table and Column Design + + Row enrichment + +The table enriches the row with the following required [common fields](https://tailpipe.io/docs/manage/table#common-fields): +- `tp_date` - The date the event was originally generated. +- `tp_id` - A unique identifier for the row. In Turbot plugins, typically set to an [xid](https://github.com/rs/xid). +- `tp_index` - The index used to partition the data, e.g., AWS account ID, GitHub organization, hostname. +- `tp_ingest_timestamp` - The timestamp when the event was ingested into the system. +- `tp_timestamp` - The timestamp when the event was originally generated. + +Additional common fields, like `tp_ips` and `tp_source_location` should be added based on what is available in the log type. + + Default file layout + +For each supported artifact source, the table defines a default `FileLayout` that uses the most common pattern. For instance, the `aws_cloudtrail_log` uses the standard AWS log format as the default `FileLayout` for the `aws_s3_bucket` source as part of its [GetSourceMetadata function](https://github.com/turbot/tailpipe-plugin-aws/blob/6b2620d49330aff84f8879e2740fabb39fc87b79/tables/cloudtrail_log/cloudtrail_log_table.go#L26-L28). + +### Logging + + Error info + +When the plugin returns an error, it includes the location and any related args, along with the error itself. + +### Column Types + + Struct vs. JSON + +Struct columns are preferred, so columns with nested properties should use the `STRUCT` type if the property structure is known; however, if a column has unknown properties, it should use the `JSON` type. + +## Documentation + +### Index Documentation + + Front matter + +The index document contains a front matter block, like the one below: + +```yml +--- +organization: Turbot +category: ["public cloud"] +icon_url: "/images/plugins/turbot/aws.svg" +brand_color: "#FF9900" +display_name: Amazon Web Services +description: "Tailpipe plugin for collecting and querying various logs from AWS." +og_description: "Collect AWS logs and query them instantly with SQL! Open source CLI. No DB required." +--- +``` + Front matter: category + +The category is an appropriate choice from the list at [hub.tailpipe.io/plugins](https://hub.tailpipe.io/plugins). + + Front matter: icon_url + +The icon URL is a link to an `.svg` file hosted on hub.tailpipe.io. Please request an icon through the [Turbot Community Slack](https://turbot.com/community/join) and a URL will be provided to use in this variable. + + Front matter: brand color + +The color matches the provider's brand guidelines, typically stated on a page like [this one](https://developer.amazon.com/en-US/alexa/branding/alexa-guidelines/brand-guidelines/color) for Amazon Alexa. + + Plugin description + +The description in `docs/index.md` is appropriate for the provider. The [AWS plugin](https://hub.tailpipe.io/plugins/turbot/aws), for example, uses: + +``` +AWS provides on-demand cloud computing platforms and APIs to authenticated customers on a metered pay-as-you-go basis. +``` + +The opening sentence of the Wikipedia page for the provider can be a good source of guidance here. + + Powerpipe mods + +If a plugin has at least one table with a matching Detections mod, e.g., [AWS CloudTrail Log Detections](https://hub.powerpipe.io/mods/turbot/tailpipe-mod-aws-cloudtrail-log-detections), then the `Detections as Code with Powerpipe` section is added and links to the mod. + + Connection credentials + +If a plugin defines a connection type, the plugin should: + +- List connection arguments +- Link to provider documentation +- Explain how to use existing CLI creds when that's possible + +### Table Documentation + + Example usage + +Each table document shows a basic example of how to configure a partition with a connection (if applicable) and then collect log data from that partition. + + Useful examples + +Each table document shows 3-5 useful examples that reflect real-world scenarios. + + Column specificity + +All examples should specify columns and should usually include the log's timestamp and actor (if available) as the first column. Using `SELECT *` is generally not preferred as it can produce too much data to be helpful. + +### Source Documentation + + Example configurations + +Each source document shows 3-5 configuration examples that users can copy, modify, and use to collect logs easily. + + Arguments + +All source arguments should be listed along with their types and descriptions, and if they're required. + + Table defaults + +For each table that defines defaults for the source's arguments, a link should be added to the relevant table's `Source Defaults` section. For instance, the [aws_s3_bucket](https://hub.tailpipe.io/plugins/turbot/aws/sources/aws_s3_bucket#table-defaults) document links to [aws_cloudtrail_log - Source Defaults: aws_s3_bucket](https://hub.tailpipe.io/plugins/turbot/aws/tables/aws_cloudtrail_log#aws_s3_bucket). + +## Final Review + + Testing + +The plugin has been tested with real log data. + + Matching query examples + +The example in `README.md` matches the one in `docs/index.md`. + + Icon + +Each plugin has an icon shown on the Hub. Please request an icon through the [Turbot Community Slack](https://tailpipe.io/community/join). + + Ease of first use + +The plugin really nails easy setup, there's a short path to a first successful query, and it runs quickly. + + Pre-mortem + +You've considered, and addressed, reasons why this plugin could fail to delight its community. diff --git a/docs/sidebar.json b/docs/sidebar.json index ab853d1..2e75590 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -90,6 +90,13 @@ "reference/config-files/workspace" ] }, + { + "type": "category", + "id": "develop", + "label": "Developers", + "link": "reference/develop", + "items": ["reference/develop/plugin-release-checklist"] + }, "reference/glossary" ] }, @@ -104,5 +111,4 @@ "pipes-ecosystem/pipes" ] } - -] \ No newline at end of file +]