Skip to content

Commit

Permalink
Refactor pipelines commands into separate files, rename some options
Browse files Browse the repository at this point in the history
**Pipelines is currently in closed beta, renaming these without
providing aliases _should_ have no impact**

Moving each of the subcommand handlers and options for each subcommand into its own
file. This also renames many of the parameters to be more specific.

The following parameters have been renamed:

| Previous Name | New Name |
| ---- | ---- |
| access-key-id | r2-access-key-id |
| secret-access-key | r2-secret-access-key |
| transform | transform-worker |
| r2 | r2-bucket |
| prefix | r2-prefix |
| binding | enable-worker-binding |
| http | enable-http |
| authentication | require-http-auth |
| filename | file-template |
| filepath | partition-template |

Adds the following new option for `create` and `update` commands:

```
--cors-origins           CORS origin allowlist for HTTP endpoint (use * for any origin)  [array]
```

Closes https://jira.cfdata.org/browse/PIPE-160.
  • Loading branch information
cmackenzie1 committed Jan 14, 2025
1 parent 19228e5 commit e7987d4
Show file tree
Hide file tree
Showing 10 changed files with 845 additions and 474 deletions.
26 changes: 26 additions & 0 deletions .changeset/short-kids-hunt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
"wrangler": minor
---

Rename wrangler pipelines <create|update> flags, add `--cors-origins` flag

The following parameters have been renamed:

| Previous Name | New Name |
| ----------------- | --------------------- |
| access-key-id | r2-access-key-id |
| secret-access-key | r2-secret-access-key |
| transform | transform-worker |
| r2 | r2-bucket |
| prefix | r2-prefix |
| binding | enable-worker-binding |
| http | enable-http |
| authentication | require-http-auth |
| filename | file-template |
| filepath | partition-template |

Adds the following new option for `create` and `update` commands:

```
--cors-origins CORS origin allowlist for HTTP endpoint (use * for any origin) [array]
```
62 changes: 32 additions & 30 deletions packages/wrangler/src/__tests__/pipelines.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -283,47 +283,49 @@ describe("pipelines", () => {
POSITIONALS
pipeline The name of the new pipeline [string] [required]
Source settings:
--enable-worker-binding Send data from a Worker to a Pipeline using a Binding [boolean] [default: true]
--enable-http Generate an endpoint to ingest data via HTTP [boolean] [default: true]
--require-http-auth Require Cloudflare API Token for HTTPS endpoint authentication [boolean] [default: false]
--cors-origins CORS origin allowlist for HTTP endpoint (use * for any origin) [array]
Batch hints:
--batch-max-mb Maximum batch size in megabytes before flushing [number]
--batch-max-rows Maximum number of rows per batch before flushing [number]
--batch-max-seconds Maximum age of batch in seconds before flushing [number]
Transformations:
--transform-worker PipelineTransform worker and entrypoint (<worker>.<entrypoint>) [string]
Destination settings:
--r2-bucket Destination R2 bucket name [string] [required]
--r2-access-key-id R2 service Access Key ID for authentication. Leave empty for OAuth confirmation. [string]
--r2-secret-access-key R2 service Secret Access Key for authentication. Leave empty for OAuth confirmation. [string]
--r2-prefix Prefix for storing files in the destination bucket [string] [default: \\"\\"]
--compression Compression format for output files [string] [choices: \\"none\\", \\"gzip\\", \\"deflate\\"] [default: \\"gzip\\"]
--file-template Template for individual file names (must include \${slug}) [string] [default: \\"\${slug}\${extension}\\"]
--partition-template Path template for partitioned files in the bucket [string] [default: \\"event_date=\${date}/hr=\${hr}\\"]
GLOBAL FLAGS
-c, --config Path to Wrangler configuration file [string]
-e, --env Environment to use for operations and .env files [string]
-h, --help Show help [boolean]
-v, --version Show version number [boolean]
OPTIONS
--secret-access-key The R2 service token Access Key to write data [string]
--access-key-id The R2 service token Secret Key to write data [string]
--batch-max-mb The approximate maximum size (in megabytes) for each batch before flushing (range: 1 - 100) [number]
--batch-max-rows The approximate maximum number of rows in a batch before flushing (range: 100 - 1000000) [number]
--batch-max-seconds The approximate maximum age (in seconds) of a batch before flushing (range: 1 - 300) [number]
--transform The worker and entrypoint of the PipelineTransform implementation in the format \\"worker.entrypoint\\"
Default: No transformation worker [string]
--compression Sets the compression format of output files
Default: gzip [string] [choices: \\"none\\", \\"gzip\\", \\"deflate\\"]
--prefix Optional base path to store files in the destination bucket
Default: (none) [string]
--filepath The path to store partitioned files in the destination bucket
Default: event_date=\${date}/hr=\${hr} [string]
--filename The name of each unique file in the bucket. Must contain \\"\${slug}\\". File extension is optional
Default: \${slug}\${extension} [string]
--binding Enable Worker binding to this pipeline [boolean] [default: true]
--http Enable HTTPS endpoint to send data to this pipeline [boolean] [default: true]
--authentication Require authentication (Cloudflare API Token) to send data to the HTTPS endpoint [boolean] [default: false]
--r2 Destination R2 bucket name [string] [required]"
-v, --version Show version number [boolean]"
`);
});

it("should create a pipeline with explicit credentials", async () => {
const requests = mockCreateRequest("my-pipeline");
await runWrangler(
"pipelines create my-pipeline --r2 test-bucket --access-key-id my-key --secret-access-key my-secret"
"pipelines create my-pipeline --r2-bucket test-bucket --r2-access-key-id my-key --r2-secret-access-key my-secret"
);
expect(requests.count).toEqual(1);
});

it("should fail a missing bucket", async () => {
const requests = mockCreateR2TokenFailure("bad-bucket");
await expect(
runWrangler("pipelines create bad-pipeline --r2 bad-bucket")
runWrangler("pipelines create bad-pipeline --r2-bucket bad-bucket")
).rejects.toThrowError();

await endEventLoop();
Expand All @@ -338,7 +340,7 @@ describe("pipelines", () => {
it("should create a pipeline with auth", async () => {
const requests = mockCreateRequest("my-pipeline");
await runWrangler(
"pipelines create my-pipeline --authentication --r2 test-bucket --access-key-id my-key --secret-access-key my-secret"
"pipelines create my-pipeline --require-http-auth --r2-bucket test-bucket --r2-access-key-id my-key --r2-secret-access-key my-secret"
);
expect(requests.count).toEqual(1);

Expand All @@ -352,7 +354,7 @@ describe("pipelines", () => {
it("should create a pipeline without http", async () => {
const requests = mockCreateRequest("my-pipeline");
await runWrangler(
"pipelines create my-pipeline --http=false --r2 test-bucket --access-key-id my-key --secret-access-key my-secret"
"pipelines create my-pipeline --enable-http=false --r2-bucket test-bucket --r2-access-key-id my-key --r2-secret-access-key my-secret"
);
expect(requests.count).toEqual(1);

Expand Down Expand Up @@ -475,7 +477,7 @@ describe("pipelines", () => {
const updateReq = mockUpdateRequest(update.name, update);

await runWrangler(
"pipelines update my-pipeline --r2 new-bucket --access-key-id service-token-id --secret-access-key my-secret-access-key"
"pipelines update my-pipeline --r2-bucket new-bucket --r2-access-key-id service-token-id --r2-secret-access-key my-secret-access-key"
);

expect(updateReq.count).toEqual(1);
Expand All @@ -495,7 +497,7 @@ describe("pipelines", () => {
const updateReq = mockUpdateRequest(update.name, update);

await runWrangler(
"pipelines update my-pipeline --r2 new-bucket --access-key-id new-key --secret-access-key new-secret"
"pipelines update my-pipeline --r2-bucket new-bucket --r2-access-key-id new-key --r2-secret-access-key new-secret"
);

expect(updateReq.count).toEqual(1);
Expand All @@ -516,7 +518,7 @@ describe("pipelines", () => {
const updateReq = mockUpdateRequest(update.name, update);

await runWrangler(
"pipelines update my-pipeline --binding=false --http --authentication"
"pipelines update my-pipeline --enable-worker-binding=false --enable-http --require-http-auth"
);

expect(updateReq.count).toEqual(1);
Expand All @@ -534,7 +536,7 @@ describe("pipelines", () => {
});
await expect(
runWrangler(
"pipelines update bad-pipeline --r2 new-bucket --access-key-id new-key --secret-access-key new-secret"
"pipelines update bad-pipeline --r2-bucket new-bucket --r2-access-key-id new-key --r2-secret-access-key new-secret"
)
).rejects.toThrowError();

Expand Down
Loading

0 comments on commit e7987d4

Please sign in to comment.