Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add instructions for MSK using SASL/SCRAM #97

Merged
merged 3 commits into from
Aug 31, 2023
Merged

Add instructions for MSK using SASL/SCRAM #97

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5988a86
Add instructions for MSK using SASL/SCRAM
vordimous Aug 30, 2023
68604b7
update ports
vordimous Aug 30, 2023
080200a
clarify plaintext and key
vordimous Aug 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 27 additions & 9 deletions src/how-tos/amazon-msk/production.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Check out the [Troubleshooting](../../reference/troubleshooting/amazon-msk.md) g

Before setting up internet access to your MSK Cluster, you will need the following:

* an MSK Cluster configured for TLS encrypted client access
* an MSK Cluster configured for SASL/SCRAM authentication
* an VPC security group for MSK Proxy instances
* an IAM security role for MSK Proxy instances
* subscription to Zilla Plus (Public MSK Proxy) via AWS Marketplace
Expand All @@ -59,6 +59,20 @@ Then follow the [Create MSK Cluster](../../reference/amazon-msk/create-msk-clust
Name: `aklivity`\
VPC: `my-msk-cluster`\
Subnets: `my-msk-cluster-1a` `my-msk-cluster-1b` `my-msk-cluster-1c`
Access control methods: `SASL/SCRAM authentication`

Review and create the MSK Cluster.

When the MSK cluster is created you will need to follow the [Sign-in credentials authentication with AWS Secrets Manager](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html) to associate your `AmazonMSK_*` secret to your cluster. There will be a prompt on the cluster summary page to create a new secret or associate an existing one. For the remainder of this doc we will assume the following values for this secret:

Name: `AmazonMSK_alice`\
Use `Plaintext` value:

```json:no-line-numbers
{"username":"alice","password":"alice-secret"}
```

Encryption key: `<Customer managed key>`

::: tip
This creates your MSK cluster in preparation for secure access via the internet.
Expand All @@ -77,7 +91,7 @@ Description: Kafka clients and SSH access
### Inbound Rule

Type: `Custom TCP`\
Port: `9094`\
Port: `9096`\
Source: `<Any IPv4>`

### Inbound Rule
Expand All @@ -99,7 +113,7 @@ Security Group: `default` `(MSK security group)`
### Inbound Rule

Type: `Custom TCP`\
Port: `9094`\
Port: `9096`\
Source: `Custom Security groups`: `my-msk-proxy`

::: tip
Expand Down Expand Up @@ -193,7 +207,7 @@ Subnets: `my-msk-cluster-1a` `my-msk-cluster-1b` `my-msk-cluster-1c`
#### MSK Configuration

Wildcard DNS pattern [1]: `*.aklivity.[...].amazonaws.com`\
Port number: `9094`
Port number: `9096`

#### MSK Proxy Configuration

Expand All @@ -203,7 +217,7 @@ Role: `aklivity-public-msk-proxy`\
Security Groups: `my-msk-proxy`\
Secrets Manager Secret ARN [3]: [`<LetsEncrypt signed certificate's private key secret ARN>`](../../reference/amazon-msk/create-server-certificate-letsencrypt.md)
Public Wildcard DNS [4]: `*.example.aklivity.io`\
Public Port: `9094`\
Public Port: `9096`\
Key pair for SSH access [5]: `<key pair>`

### Step3. Configure stack options: `(defaults)`
Expand Down Expand Up @@ -326,14 +340,18 @@ We use a generic Kafka client here, however the setup for any Kafka client, incl

With the Kaka client now installed we are ready to configure it and point it at the Public MSK Proxy.

The MSK Proxy relies on TLS so we need to create a file called `client.properties` that tells the Kafka client to use SSL as the security protocol.
The MSK Proxy relies on encrypted SASL/SCRAM so we need to create a file called `client.properties` that tells the Kafka client to use SASL_SSL as the security protocol with SCRAM-SHA-512 encryption.

Notice we used the default username and password, but you will need to replace those with your own credentials from the `AmazonMSK_*` secret you created.

::: code-tabs#shell

@tab client.properties

```toml:no-line-numbers
security.protocol=SSL
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="alice" password="alice-secret";
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
```

:::
Expand All @@ -349,7 +367,7 @@ We can now verify that the Kafka client can successfully communicate with your M
If using the wildcard DNS pattern `*.example.aklivity.io`, then we use the following as TLS bootstrap server names for the Kafka client:

```text:no-line-numbers
b-1.example.aklivity.io:9094,b-2.example.aklivity.io:9094,b-3.example.aklivity.io:9094
b-1.example.aklivity.io:9096,b-2.example.aklivity.io:9096,b-3.example.aklivity.io:9096
```

::: warning
Expand All @@ -358,7 +376,7 @@ Replace these TLS bootstrap server names accordingly for your own custom wildcar

#### Create a Topic

Use the Kafka client to create a topic called `public-proxy-test`, replacing`<tls-bootstrap-server-names>` **** in the command below with the TLS proxy names of your Public MSK Proxy:
Use the Kafka client to create a topic called `public-proxy-test`, replacing `<tls-bootstrap-server-names>` **** in the command below with the TLS proxy names of your Public MSK Proxy:

```bash:no-line-numbers
bin/kafka-topics.sh --create --topic public-proxy-test --partitions 3 --replication-factor 3 --command-config client.properties --bootstrap-server <tls-bootstrap-server-names>
Expand Down
Loading