-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
- Loading branch information
1 parent
ec26594
commit 0ffcae5
Showing
3 changed files
with
85 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,35 +1,66 @@ | ||
|
||
# Introduction | ||
|
||
|
||
|
||
# Transformations | ||
|
||
|
||
## Archive | ||
|
||
The Archive transformation is used to help preserve all of the data for a message when archived to S3. | ||
|
||
|
||
|
||
|
||
### Note | ||
|
||
This transform works by copying the key, value, topic, and timestamp to new record where this is all contained in the value of the message. This will allow connectors like Confluent's S3 connector to properly archive the record. | ||
|
||
|
||
### Configuration | ||
|
||
| Name | Type | Importance | Default Value | Validator | Documentation| | ||
| ---- | ---- | ---------- | ------------- | --------- | -------------| | ||
|
||
#### Examples | ||
|
||
##### Standalone Example | ||
|
||
#### Standalone Example | ||
This configuration is used typically along with [standalone mode](http://docs.confluent.io/current/connect/concepts.html#standalone-workers). | ||
|
||
```properties | ||
transforms=Archive | ||
transforms.Archive.type=com.github.jcustenborder.kafka.connect.archive.Archive | ||
# The following values must be configured. | ||
name=Connector1 | ||
connector.class=org.apache.kafka.some.SourceConnector | ||
tasks.max=1 | ||
transforms=tran | ||
transforms.tran.type=com.github.jcustenborder.kafka.connect.archive.Archive | ||
``` | ||
|
||
#### Distributed Example | ||
##### Distributed Example | ||
|
||
This configuration is used typically along with [distributed mode](http://docs.confluent.io/current/connect/concepts.html#distributed-workers). | ||
Write the following json to `connector.json`, configure all of the required values, and use the command below to | ||
post the configuration to one the distributed connect worker(s). | ||
|
||
```json | ||
{ | ||
"name": "connector1", | ||
"config": { | ||
"connector.class": "com.github.jcustenborder.kafka.connect.archive.Archive", | ||
"transforms": "Archive", | ||
"transforms.Archive.type": "com.github.jcustenborder.kafka.connect.archive.Archive", | ||
} | ||
"name" : "Connector1", | ||
"connector.class" : "org.apache.kafka.some.SourceConnector", | ||
"transforms" : "tran", | ||
"transforms.tran.type" : "com.github.jcustenborder.kafka.connect.archive.Archive" | ||
} | ||
``` | ||
|
||
Use curl to post the configuration to one of the Kafka Connect Workers. Change `http://localhost:8083/` the the endpoint of | ||
one of your Kafka Connect worker(s). | ||
|
||
Create a new instance. | ||
```bash | ||
curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors | ||
``` | ||
|
||
Update an existing instance. | ||
```bash | ||
curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/TestSinkConnector1/config | ||
``` | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters