Skip to content

Commit

Permalink
Merge pull request #703 from aws-quickstart/feature/apache-airflow
Browse files Browse the repository at this point in the history 
Apache Airflow update: Removed RDS for now, added doc and refactored codes
  • Loading branch information
@shapirov103
shapirov103 authored May 27, 2023
2 parents 78dfee5 + 9689dbb commit f794509
Show file tree
Hide file tree
Showing 4 changed files with 429 additions and 4 deletions.
103 changes: 103 additions & 0 deletions docs/addons/apache-airflow.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Apache Airflow Add-on

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow contains extensible framework that allows for building workflows connecting with many technologies. For more information on Airflow, please consult the [official documentation](https://airflow.apache.org/docs/).

This add-on is an implementation of Apache Airflow on EKS using the official helm chart.

## Prerequisites

1. If you are using an S3 bucket (for logging storage) or EFS File System (for DAGs persistent storage), you must register the resources using Resource Provider. The framework provides S3 and EFS Resource Provider.
2. If you are using an EFS File System, you must include an EFS CSI Driver in the add-on array, otherwise will run into the following error: `Missing a dependency: EfsCsiDriverAddOn. Please add it to your list of addons.`.
3. If you are using an Ingress with AWS Load Balancer, you must include an AWS Load Balacner Controller in the add-on array, otherwise you will run into the following error: `Missing a dependency: AwsLoadBalancerControllerAddOn. Please add it to your list of addons`.

## Usage

```typescript
import * as cdk from 'aws-cdk-lib';
import * as blueprints from '@aws-quickstart/eks-blueprints';

const app = new cdk.App();

const apacheAirflowS3Bucket = new blueprints.CreateS3BucketProvider({
id: 'apache-airflow-s3-bucket-id',
s3BucketProps: { removalPolicy: cdk.RemovalPolicy.DESTROY }
});
const apacheAirflowEfs = new blueprints.CreateEfsFileSystemProvider({
name: 'blueprints-apache-airflow-efs',
});

const addOn = [new blueprints.EfsCsiDriverAddOn(),
new blueprints.ApacheAirflowAddOn({
enableLogging: true,
s3Bucket: 'airflow-logging-s3-bucket',
enableEfs: true,
efsFileSystem: 'apache-airflow-efs-provider',
})
];

const blueprint = blueprints.EksBlueprint.builder()
.resourceProvider('apache-airflow-s3-bucket-provider', apacheAirflowS3Bucket)
.resourceProvider('apache-airflow-efs-provider', apacheAirflowEfs)
.addOns(addOn)
.build(app, 'my-stack-name');
```

## Validation

To validate that the Airflow add-on is installed properly, ensure that the required kubernetes resources are running in the cluster:

```bash
kubectl get all -n airflow
```

```bash
NAME READY STATUS RESTARTS AGE
pod/blueprints-addon-apache-airflow-postgresql-0 1/1 Running 0 1m4s
pod/blueprints-addon-apache-airflow-scheduler-697958497d-tbblk 2/2 Running 0 1m4s
pod/blueprints-addon-apache-airflow-statsd-5b97b9fcb4-9r8qc 1/1 Running 0 1m4s
pod/blueprints-addon-apache-airflow-triggerer-86b94646c6-xrjnn 2/2 Running 0 1m4s
pod/blueprints-addon-apache-airflow-webserver-6b8db695fc-9d87q 1/1 Running 0 1m4s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/blueprints-addon-apache-airflow-postgresql ClusterIP 172.20.155.11 <none> 5432/TCP 1m4s
service/blueprints-addon-apache-airflow-postgresql-hl ClusterIP None <none> 5432/TCP 1m4s
service/blueprints-addon-apache-airflow-statsd ClusterIP 172.20.247.149 <none> 9125/UDP,9102/TCP 1m4s
service/blueprints-addon-apache-airflow-webserver ClusterIP 172.20.211.66 <none> 8080/TCP 1m4s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/blueprints-addon-apache-airflow-scheduler 1/1 1 1 1m4s
deployment.apps/blueprints-addon-apache-airflow-statsd 1/1 1 1 1m4s
deployment.apps/blueprints-addon-apache-airflow-triggerer 1/1 1 1 1m4s
deployment.apps/blueprints-addon-apache-airflow-webserver 1/1 1 1 1m4s

NAME DESIRED CURRENT READY AGE
replicaset.apps/blueprints-addon-apache-airflow-scheduler-669dff9c6d 0 0 0 1m4s
replicaset.apps/blueprints-addon-apache-airflow-scheduler-697958497d 1 1 1 1m4s
replicaset.apps/blueprints-addon-apache-airflow-statsd-5b97b9fcb4 1 1 1 1m4s
replicaset.apps/blueprints-addon-apache-airflow-triggerer-7b7c69486d 0 0 0 1m4s
replicaset.apps/blueprints-addon-apache-airflow-triggerer-86b94646c6 1 1 1 1m4s
replicaset.apps/blueprints-addon-apache-airflow-webserver-5db49dcb94 0 0 0 1m4s
replicaset.apps/blueprints-addon-apache-airflow-webserver-6b8db695fc 1 1 1 1m4s

NAME READY AGE
statefulset.apps/blueprints-addon-apache-airflow-postgresql 1/1 1m4s
```

## Testing

To test the Airflow functionality, expose the webserver by port forwarding:

```bash
kubectl port-forward service/blueprints-addon-apache-airflow-webserver -n airflow 8080:8080
```

You should be able to access the webserver via your browser: `http://localhost:8080`. Default username and password are both `admin`.

## Functionality

Applies the Apache Airflow add-on to an Amazon EKS cluster. Optionally, you can leverage integrations with the following AWS services out of the box:

1. S3 Bucket for storing server, worker, and scheduler logs remotely.
2. EFS File System for storing DAGs
3. AWS Load Balancer for ingress to the webserver
4. A Certificate for AWS Certificate Manager for SSL
20 changes: 16 additions & 4 deletions examples/blueprint-construct/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@ export default class BlueprintConstruct {
const ampWorkspaceName = "blueprints-amp-workspace";
const ampWorkspace: CfnWorkspace = blueprints.getNamedResource(ampWorkspaceName);

const apacheAirflowS3Bucket = new blueprints.CreateS3BucketProvider({
id: 'apache-airflow-s3-bucket-id',
s3BucketProps: { removalPolicy: cdk.RemovalPolicy.DESTROY }
});
const apacheAirflowEfs = new blueprints.CreateEfsFileSystemProvider({
name: 'blueprints-apache-airflow-efs',
});

const addOns: Array<blueprints.ClusterAddOn> = [
new blueprints.addons.AwsLoadBalancerControllerAddOn(),
new blueprints.addons.AppMeshAddOn(),
Expand Down Expand Up @@ -156,6 +164,12 @@ export default class BlueprintConstruct {
new blueprints.AwsForFluentBitAddOn(),
new blueprints.FluxCDAddOn(),
new blueprints.GrafanaOperatorAddon(),
new blueprints.ApacheAirflowAddOn({
enableLogging: true,
s3Bucket: 'apache-airflow-s3-bucket-provider',
enableEfs: true,
efsFileSystem: 'apache-airflow-efs-provider'
})
];

// Instantiated to for helm version check.
Expand Down Expand Up @@ -280,10 +294,8 @@ export default class BlueprintConstruct {
secondarySubnetCidrs: ["100.64.0.0/24","100.64.1.0/24","100.64.2.0/24"]
}))
.resourceProvider("node-role", nodeRole)
.resourceProvider('blueprint-s3', new blueprints.CreateS3BucketProvider({
id: 'blueprints-s3-bucket-id',
s3BucketProps: { removalPolicy: cdk.RemovalPolicy.DESTROY }
}))
.resourceProvider('apache-airflow-s3-bucket-provider', apacheAirflowS3Bucket)
.resourceProvider('apache-airflow-efs-provider', apacheAirflowEfs)
.clusterProvider(clusterProvider)
.resourceProvider(ampWorkspaceName, new blueprints.CreateAmpProvider(ampWorkspaceName, ampWorkspaceName))
.teams(...teams, new blueprints.EmrEksTeam(dataTeam), new blueprints.BatchEksTeam(batchTeam))
Expand Down
Loading

0 comments on commit f794509

Please sign in to comment.