Merge pull request #38 from turbot/updates-10

proofread
turbot · Jan 30, 2025 · d0a2952 · d0a2952
2 parents 35900ff + 88ce757
commit d0a2952
Show file tree

Hide file tree

Showing 16 changed files with 86 additions and 104 deletions.
diff --git a/docs/learn.md b/docs/learn.md
@@ -69,8 +69,7 @@ Create a file, e.g. `~/.tailpipe/config/aws.tpc`, with a `connection` and `parti
 > ```hcl
 > partition "aws_cloudtrail_log" "flaws" {
 > source "file" {
->    paths       = ["/Users/dboeke/flaws/flaws_cloudtrail_logs"]
->    file_layout = "%{DATA}.json.gz"
+    paths       = ["/Users/dboeke/flaws/flaws_cloudtrail_logs"]
 >  }
 >}
 >```
@@ -84,21 +83,21 @@ tailpipe collect aws_cloudtrail_log
 ```
 
 
-Tailpipe will download the files from the source, decompress and parse them, and add the data to the Tailpipe database in the [standard hive file structure](/docs/reference/glossary#hive).
+Tailpipe will download the files from the source, decompress and parse them, and add the data to the Tailpipe database in the [standard hive file structure](/docs/manage/hive).
 
 ![](/learn/collection.png)
 
-To see the table that was created:
-
-```bash
-$ tailpipe table list
-NAME                  PLUGIN                                       LOCAL SIZE    FILES    ROWS
-aws_cloudtrail_log    hub.tailpipe.io/plugins/turbot/aws@latest    42 MB         2        160,581
-```
 
 ## Query your logs
 
-Tailpipe provides an interactive SQL shell for analyzing your collected data. Run `tailpipe query` to start the query shell.
+Tailpipe provides an interactive SQL shell for analyzing your collected data. Run `tailpipe query` to start the query shell. To see the table that was created:
+
+```bash
+$ tailpipe query
+> .inspect
+Table              Plugin          
+aws_cloudtrail_log [email protected]
+```
 
 You can count the records in the table:
 

diff --git a/docs/manage/collection.md b/docs/manage/collection.md
@@ -26,7 +26,7 @@ Collect a specific partition.
 tailpipe collect aws_cloudtrail_log.dev
 ```
 
-See [/docs/reference/cli/collect](/docs/reference/cli/collect) for more examples.
+See [collect](/docs/reference/cli/collect) for more examples.
 
 
 The collection process always writes to a local [workspace](/docs/manage/workspace), and does so on a per-partition basis.  While you may specify multiple partitions on the command line, `partition` is the unit of collection.
@@ -51,7 +51,7 @@ tailpipe collect aws_cloudtrail_log.test --from T-180d
 tailpipe collect aws_cloudtrail_log.test --from 2024-01-01
 ```
 
-- Subsequent collection runs occur chronologically resuming from the last collection by default, so there are no time gaps while the data is being collected.
+Subsequent collection runs occur chronologically resuming from the last collection by default, so there are no time gaps while the data is being collected.
 
 <!--
 - The data is available for querying even while partition collection is still occurring.

diff --git a/docs/manage/plugin.md b/docs/manage/plugin.md
@@ -19,8 +19,6 @@ This will download the latest aws plugin from the hub registry, and will set up
 > [!NOTE]
 > If you install multiple versions of a plugin only the first installation will create a connection automatically for you, you will need to create/edit a [connection](/docs/manage/connection) configuration file in order to use the additional versions of the plugin. 
 
-> ^ Will this apply to tailpipe?
-
 ## Installing a Specific Version
 
 To install a specific version, simply specify the version tag after the plugin name, separated by `@` or `:`
@@ -108,9 +106,6 @@ For example, consider a `myplugin` plugin that you have developed.  To install i
     }
     ```
 
-> [!WARNING]
-> ^ Keep? If so, change how?
-
 
 ## Installing Missing Plugins
 
@@ -134,15 +129,15 @@ tail
 ## Viewing Installed Plugins
 You can list the installed plugins with the `tailpipe plugin list` command:
 
-```
+```hcl
 $ tailpipe plugin list
 ┌─────────────────────────────────────────────────────┬─────────┬────────────────────────────────────────────┐
 │ NAME                                                │ VERSION │ CONNECTIONS                                │
 ├─────────────────────────────────────────────────────┼─────────┼────────────────────────────────────────────┤
-│ hub.tailpipe.io/plugins/turbot/aws@latest           │ 0.1.0   │ aws,aws_account_aaa,aws_account_aab         │
-│ hub.tailpipe.io/plugins/turbot/gcp@latest           │ 0.0.6   │ gcp_project_a,gcp,gcp_project_b             │
-│ hub.tailpipe.io/plugins/turbot/github@latest        │ 0.0.5   │ github                                      │
-└─────────────────────────────────────────────────────┴─────────┴─────────────────────────────────────────────┘
+│ hub.tailpipe.io/plugins/turbot/aws@latest           │ 0.1.0   │ aws,aws_account_aaa,aws_account_aab        │
+│ hub.tailpipe.io/plugins/turbot/gcp@latest           │ 0.0.6   │ gcp_project_a,gcp,gcp_project_b            │
+│ hub.tailpipe.io/plugins/turbot/github@latest        │ 0.0.5   │ github                                     │
+└─────────────────────────────────────────────────────┴─────────┴────────────────────────────────────────────┘
 ```
 
 ## Updating Plugins
@@ -192,7 +187,7 @@ $ tailpipe plugin list
 |                    NAME                    | VERSION | CONNECTIONS |
 +--------------------------------------------+---------+-------------+
 | hub.tailpipe.io/plugins/turbot/aws@latest  | 0.0.5   | aws         |
-    +--------------------------------------------+---------+-------------+
++--------------------------------------------+---------+-------------+
 ```
 
 ## Tailpipe Plugin Registry Support Lifecycle

diff --git a/docs/manage/source.md b/docs/manage/source.md
@@ -27,12 +27,10 @@ partition "aws_cloudtrail_log" "test" {
   source "aws_s3_bucket" {
     connection    = connection.aws.logs
     bucket        = "my-logs-bucket"
-    file_layout   = "{%DATA}.json.gz"
   }
   
   source "file" {
     path       = "/path/to/files"
-    file_layout   = "{%DATA}.json.gz"
   }
 }
 ```

diff --git a/docs/pipes-ecosystem/index.md b/docs/pipes-ecosystem/index.md
@@ -10,6 +10,6 @@ Tailpipe is a part of a larger ecosystem of products that provide a flexible pla
 
 - [Steampipe](/docs/pipes-ecosystem/steampipe) provides a dynamic cloud inventory, allowing you to to query and analyze data from multiple sources via SQL.  Its plugin-based architecture ensures extensibility, enabling connections to APIs, cloud services, databases, and more.  
 
-- [Flowpipe](/docs/pipes-ecosystem/flowpipe) further enriches the platform with workflow orchestration.  Flowpipe allows you to create "pipelines as code" to define workflows and other tasks that run in a sequence.  With Flowpipe, you can turn your Steampipe and Tailpipe ;insights into into action!
+- [Flowpipe](/docs/pipes-ecosystem/flowpipe) further enriches the platform with workflow orchestration.  Flowpipe allows you to create "pipelines as code" to define workflows and other tasks that run in a sequence.  With Flowpipe, you can turn your Steampipe and Tailpipe insights into into action!
 
 - [Turbot Pipes](/docs/pipes-ecosystem/pipes) provides a hosted cloud intelligence, automation & security platform built specifically for DevOps. Pipes provides a managed Steampipe database instances, shared Powerpipe dashboards & snapshots, Flowpipe triggers & pipelines, and more!
diff --git a/docs/reference/cli/compact.md b/docs/reference/cli/compact.md
@@ -2,10 +2,10 @@
 title: tailpipe compact
 ---
 
-Compact multiple Parquet files per day to one per day.
-
 # tailpipe compact
 
+Compact multiple Parquet files per day to one per day.
+
 ## Usage
 ```bash
  tailpipe compact [table] [table.partition] [args]

diff --git a/docs/reference/cli/connect.md b/docs/reference/cli/connect.md
@@ -53,6 +53,9 @@ Connect to the Tailpipe database, filtered to records after the start time:
 
 ```bash
 tailpipe connect --from 2025-01-01
+```
+
+```bash
 /home/jon/.tailpipe/data/default/tailpipe_20250115140447.db
 ```
 
@@ -61,19 +64,13 @@ tailpipe connect --from 2025-01-01
 > 
 > duckdb /home/jon/.tailpipe/data/default/tailpipe_20241212134120.db
 
-```bash
-tailpipe connect --from 2025-01-01
-/home/jon/.tailpipe/data/default/tailpipe_20250115140447.db
-```
+Connect with no filter, show output as json:
 
 ```bash
-tailpipe connect --from 2025-01-01
-/home/jon/.tailpipe/data/default/tailpipe_20250115140447.db
+tailpipe connect --output json
 ```
 
-Show the filepath for the connected database:
-
 ```bash
-tailpipe connect --output json
+{"database_filepath":"/Users/jonudell/.tailpipe/data/default/tailpipe_20250129204416.db"}
 ```
 
diff --git a/docs/reference/cli/partition.md b/docs/reference/cli/partition.md
@@ -8,112 +8,112 @@ List, show, and delete Tailpipe partitions.
 
 ## Usage
 ```bash
-tailpipe partition delete table_name.partition_name [args]
 tailpipe partition list [args]
 tailpipe partition show table_name.partition_name [args]
+tailpipe partition delete table_name.partition_name [args]
 ```
 
 ## Sub-Commands
 
 | Command | Description
 |-|-
-| [delete](#tailpipe-partition-delete) | Delete a partition.
 | [list](#tailpipe-partition-list) | List all partitions.
 | [show](#tailpipe-partition-show)  | Show details of a partition.
+| [delete](#tailpipe-partition-delete) | Delete a partition.
 
-## tailpipe partition delete
-Delete a partition.
+
+## tailpipe partition list
+List all partitions.
 
 ### Arguments
 
 | Flag | Description
 |-|-
-|  `--force`     |  Force delete without confirmation
-|  `--from`      |  Specify the start time
-|  `--help`      |  Help for delete
+|  `--help`      |  Show help.
+|  `--output`    |  Output format: json, text (default)
+|  `--to`        |  Delete days older than than a relative or absolute date.
 
 ### Examples
 
-Delete an entire partition.
+List all partitions in table format.
 
 ```bash
-tailpipe partition delete aws_cloudtrail_log.dev
+tailpipe partition list
 ```
 
-Delete days newer than a relative date.
+List all partitions in `JSON` format. 
 
 ```bash
-tailpipe partition delete aws_cloudtrail_log.dev --from T-7d
+tailpipe partition list --output json
 ```
 
-Delete days newer than an absolute date with no confirmation.
+## tailpipe partition show
+Show details for a partition.
 
-```bash
-tailpipe partition delete aws_cloudtrail_log.dev --from 1/1/24 --force
-```
+Flag | Description
+|-|-
+|  `--help`      |  Show help.
+|  `--output`    |  Output format: json, text (default)
 
-Delete days older than a relative date.
+
+### Examples
+
+Show details in text format.
 
 ```bash
-tailpipe partition delete aws_cloudtrail_log.dev --to T-7d
+tailpipe partition show aws_cloudtrail-log.prod
 ```
 
-Delete days older than an absolute date.
+Show details in JSON format.
 
 ```bash
-tailpipe partition delete aws_cloudtrail_log.dev --to 1/1/24
+tailpipe partition show aws_cloudtrail-log.prod --output json
 ```
 
-## tailpipe partition list
-List all partitions.
+## tailpipe partition delete
+Delete a partition.
 
 ### Arguments
 
 | Flag | Description
 |-|-
-|  `--help`      |  Show help.
-|  `--output`    |  Output format: json, text (default)
-|  `--to`        |  Delete days older than than a relative or absolute date.
+|  `--force`     |  Force delete without confirmation
+|  `--from`      |  Specify the start time
+|  `--help`      |  Help for delete
 
 ### Examples
 
-List partitions in table format.
+Delete an entire partition.
 
 ```bash
-tailpipe partition list
+tailpipe partition delete aws_cloudtrail_log.dev
 ```
 
-List all partitions in `JSON` format. 
+Delete days newer than a relative date.
 
 ```bash
-tailpipe partition list --output json
+tailpipe partition delete aws_cloudtrail_log.dev --from T-7d
 ```
 
-## tailpipe partition show
-Show details for a partition.
-
-Flag | Description
-|-|-
-|  `--help`      |  Show help.
-|  `--output`    |  Output format: json, text (default)
-
+Delete days newer than an absolute date with no confirmation.
 
-### Examples
+```bash
+tailpipe partition delete aws_cloudtrail_log.dev --from 2024-01-01  --force
+```
 
-Show details in text format.
+Delete days older than a relative date.
 
 ```bash
-tailpipe partition show aws_cloudtrail-log.prod
+tailpipe partition delete aws_cloudtrail_log.dev --to T-7d
 ```
 
-Show details in JSON format.
+Delete days older than an absolute date.
 
 ```bash
-tailpipe partition show aws_cloudtrail-log.prod --output json
+tailpipe partition delete aws_cloudtrail_log.dev --to 2024-01-01
 ```
 
 
-
 ## Global Flags
 
 <table>

diff --git a/docs/reference/cli/plugin.md b/docs/reference/cli/plugin.md
@@ -2,11 +2,9 @@
 title: tailpipe plugin
 ---
 
-
 # tailpipe plugin
-tailpipe plugin management
 
-Plugins extend tailpipe to work with many different services and providers. Find plugins using the public registry at [hub.tailpipe.io](https://hub.tailpipe.io).
+Tailpipe plugin management. Plugins extend tailpipe to work with many different services and providers. Find plugins using the public registry at [hub.tailpipe.io](https://hub.tailpipe.io).
 
 
 ## Usage

diff --git a/docs/reference/cli/query.md b/docs/reference/cli/query.md
@@ -4,7 +4,7 @@ title: tailpipe query
 
 # tailpipe query
 
-Query a Tailpipe table
+Query a Tailpipe table.
 
 ## Usage
 ```bash
@@ -119,7 +119,6 @@ Run the SQL command in the `my_queries/my_query.sql` file:
 tailpipe query my_queries/my_query.sql
 ```
 
-```
 Run a query and return output in CSV format:
 ```bash
 tailpipe query "select * from aws_cloudtrail_log limit 1000" --output csv

diff --git a/docs/reference/config-files/connection.md b/docs/reference/config-files/connection.md
@@ -28,9 +28,9 @@ Each connection has a type label and a name. There is a type for each service: [
 
 ### Default connections
 
-Tailpipe also creates a single **implicit default** connection for each connection type. The connection is named `default`, eg `connection.aws.default`, `connection.azure.defaut`, etc. The intent of the default connection is that Tailpipe can "just work" with your existing setup, and is similar to the default connection that Steampipe creates for each plugin. For example, the AWS default connection will use the standard AWS SDK connection resolution mechanism to create a Tailpipe connection that will use the same connection that the `aws` CLI would use.
+Tailpipe also creates a single **implicit default** connection for each connection type. The connection is named `default`, eg `connection.aws.default`, `connection.azure.default`, etc. The intent of the default connection is that Tailpipe can "just work" with your existing setup, and is similar to the default connection that Steampipe creates for each plugin. For example, the AWS default connection will use the standard AWS SDK connection resolution mechanism to create a Tailpipe connection that will use the same connection that the `aws` CLI would use.
 
-You can override the default by simply creating a connection for that type that is named `default`:
+You can override the default by creating a connection for that type that is named `default`:
 
 ```hcl
 connection "aws" "default" {