Materials for AWS CICD workshop
- Slides
- Prerequisites
- Hands-on
- Tips
- The CICD journey of SPN Infra., Coretech, Trendmico on AWS
- CI/CD for Serverless and Containerized Applications (DEV309-R1) - AWS re:Invent 2018
- https://github.com/awslabs/aws-cloudformation-templates
- A good Cloudformation examples git repo maintained by AWS
- I rely on AWS CLoudformation heavily in the following handsons for building CD
- Deep Dive into Blue/Green Deployments on AWS (DVO401) - AWS re:Invent 2015
- An AWS (Amazon Web Service) account W/ administrator permission, if you have none, to get one !
There are few things need your notices and actions in your AWS account
- All the hands-on activities will be in region us-east-1, codes are not tested in other regions.
- And you need to make sure you can create bucket and put files on S3... due to you can not use it before account activated successfully.
- Create a SSH key (e.g. cicd) in us-east-1 region under your AWS account.
- The AWS expenses will occur in the hands-ons, it should cost less than USD $1.0 for each hands-on...
- A public GitHub account, if you have none, to get one !
There are two more things need your actions in your GitHub account
- Fork this repo to under your GitHub account
- Create a valid OAuth token from GitHub, we will use it in AWS CodePipeline service.
I am trying to leverage as most managed services on AWS as I can, so I picked S3, CodePipline, CodeBuild and CloudFormation as a foundation for our hands-on.
Frankly to say, there are variety of technology combinations doing CI/CD on every environment (In this case, is AWS), so my choices here may not be the most suitable for your needs, it simply plays as a quick start to bring you the overall concepts and a PoC to home. And I hope, it can help you an idea to tailor your own one.
According to my experiences, whatever technology stacks you are using for building your CICD infra., I think there will be three aspects you should need to consider about are...
-
What CI server you are using to support your workflow ?
- In this case is CodePipline + CodeBuild
- Others may consider to use Jenkins or others CI SaaS services
-
What tools you are using to support your CD (end-to-end from your running services to monitoring) ?
- In this case is CloudFormation
- Others may write a bunch of scripts to do their CD
-
How to pass your configuration parameters (Non-secret and secret) to your CD ?
- In this case are CodePipeline + SSM parameter store
- Others may use environment variables setting supported by CI server
We use AWS CodePipeline to pull commits from GitHub, build, test and deploy a standalone VPC and a very simple API service running within it, in AWS region us-east-1.
The components are
- A standalone VPC
- public subnet * 2, private subnet * 2, NAT gateway * 2, S3 VPC endpoint * 1
- An EC2 instance as bsation node
- You can login to this VPC via this bastion node
- Application load balancer
- Two EC2 instances running a very simple web app behind the ALB
- CloudWatch Alarm
- SNS Topic for alarm mail
- CloudWatch dashboard
Click following icon to provision AWS CodePipline via AWS CloudFormation (cf-ec2-cp.yaml) in your AWS account.
- Fill up the parameters whom are empty or you want to replace
- Click next, and next again
- Check all confirmation questions for access IAM resources
- Click create ChangeSet button
- Click execute and will bring you to CloudFormation console automatically.
You can see the progress of provisioning of CodePipline on AWS CloudFormation console. After provisioning status gets to CREATE_COMPLETED, pls go to CodePipline console to find your first CodePipline project.
Pls remember to Confirm subscription of two mails sent from AWS SNS topics in your mailbox, which is the email address you had written and passed to CloudFormation.
You can see the CodePipline project starts to trigger a build automatically in few minutes. Due to the CodePipeline will pull repo at first time.
After the service provisioning gets to CREATE_COMPLETED, you can get the endpoint FQDN at the row ALBDNSName at Output tab in AWS CloudFormation console. let's veirfy your build whether successfully !
curl http://<ALBDNSName>
Welcome to my homeThe very simple API impl. is api/main.py you can take a look if interested in.
Take a look on CodePipline console and corresponding docs as follows to clarify our questions.
The action types we are using are...
| Category | Action |
|---|---|
| Source | GitHub |
| Build | AWS CodeBuild |
| Deploy | CloudFormation |
| Approval | Manual approval |
You can see more details in cf-ec2-cp.yaml
Let's say it again, this hands-on simply plays as a quick start to bring you the overall concepts and a PoC to home. And I hope, it can help you an idea to tailor your own one.
But I still can say that, CodePipeline using Jenkins need to launch a long running EC2 instance behind the scene, it is too costly for a PoC. And CodeDeploy can support variety of deployment methods (Lambda, ECS, etc), but in the end, I still need to use CloudFormation to provision monitoring related resources based on end-to-end point of view. And also, CloudFormation is more portable running in diffrent CI servers (Maybe :P).
The very simple API impl. is api/main.py you can take a look if interested in.
curl http://<ALBDNSName>/healthcheck
Hello World!This API is used by ALB and auto-scaling group, auto-scaling group will increase/decrease based on the responses of this API from every EC2 instance.
The configuration order will be ALB -> Target group -> Auto-scaling group -> Auto-scaling group configuration -> EC2. You can get start on following doc.
In this hands-on we are using Blue/Green deployment (create 2 new EC2s -> all healthchek passed -> delete 2 old EC2s), the details in following doc
Take a look on cd/cf-ec2.yaml for more details.
OpsWorks service has a lot of pitfalls and performance issues according my experiences, I recommend to use AWS::CloudFormation::Init instead.
Actually, AWS::CloudFormation::Init is based on cloud-init project.
curl http://<ALBDNSName>/sleep/20
sleep for 20 secsThis API is used to create a response latency (by seconds) of our service, to trigger the backend alarm.
- See the service alarm on CloudWatch Alarm console
- See the service dashboard on CloudWatch Dashboard
- See cd/cf-ec2.yaml for more details
In this case, the alarm will send notification to a SNS topic and then to the topic subscriber, which is, an email address you provided at first place.
The SNS Topic can send to various types of subscriber, for example, to the Slack and PagerDuty.
All these basic stuffs can be created by AWS CloudFormation
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-alarm.html
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-dashboard.html
But actually, there are still more complicated usecases the CloudFormation can not fulfill them in a out-of-box manner, in these cases, we can consider to use AWS::CloudFormation::CustomResource to deal them.
AWS::CloudFormation::CustomResource is actually a Lambda function extension plugged in CloudFormation service, which can bring a flexible way to provision and configure our resources in CloudFormation
curl http://<ALBDNSName>/secret
{"Name": "mysecret03", "Value": "mysecretonaws03"}This API is trying to demo how we pass the secret data (e.g. database password, etc) in CI/CD. In this case, we encrypted and stored the secret data in SSM parameter store at first place. And try to load and decrypt the secret data in API code.
How to use SSM parameter store
How to encrypt the data with AWS::CloudFormation::CustomResource and AWS::KMS::Key
- codepipelines/cf-ec2-cp.yaml line#302 and #324
- codepipelines/kms_encryption/main.py
How to decrypt the data in code
- api/main.py line#30
We use AWS CodePipeline to pull commits from GitHub, build, test and deploy an API gateway and multiple Lambda functions behind it, in AWS region us-east-1.
The components are
- An API gateway
- Multiple Lambda functions exposing our very simple API services
- A custom metric generated from a Lambda function logs
- CloudWatch Alarm
- SNS Topic for alarm mail
- CloudWatch dashboard
Click following icon to provision AWS CodePipline via AWS CloudFormation (cf-sl-cp.yaml) in your AWS account.
Pls go to hands-on#1-1. Launch AWS CodePipeline for reference
You can see the CodePipline project starts to trigger a build automatically in few minutes. Due to the CodePipeline will pull repo at first time.
After the service provisioning gets to CREATE_COMPLETED, you can get the endpoint FQDN at the row APIGatewayURL in Output tab of AWS CloudFormation console. let's verify your build whether run successfully !
curl <APIGatewayURL>
Welcome to my homeThe very simple API impl. is still based on api/main.py, but every API is actually constructed by Lambda functions(lambdas/), you can take a look if interested in.
You can see more details in cf-sl-cp.yaml
Pls go to hands-on#1-3. Take a look at every component in the overall CodePipeline for reference.
The very simple API impl. is still based on api/main.py, but every API is actually constructed by Lambda functions(lambdas/), you can take a look if interested in.
We are using CloudFormation W/ serverless application model(SAM) to deploy our overall resources including API gateway, Lambda functions, and others.
We basically don't need to care about the operations and scaling related issues for underlying resources due to AWS serverless architecture is taking about them for us. But the world is not perfect, there are still some issues we need to care about...for example, cold start issue, etc. Pls refer to slides,p#5 for details
curl <APIGatewayURL>/healthcheck
Hello World!This API originally used by ALB + EC2 architecture, it is useless for serverless architecture.
We are using SAM to deploy/update our API gateway and Lambda function resources, but there is actually a drawback for SAM model...which is...it replaces our underlying API gateway deployment and/or Lambda functions in place (depends on what resource you changed). It will impact our API availability in a very short of time, so you need to consider this issue whether break your SLA. Pls refer to slides,p#4 for more details.
Take a look on cd/cf-sl.yaml for more details.
UPDATE-20190320: Now API gateway supports Canary deployment to avoid previous issue. Although I still NOT implement in this handson yet, according to my experiences, I recommend you can use the SAM to implement the CD.
Pls go to hands-on#1-4.4 sleep API for reference.
I also enhanced a new testcase for this API, which is.
curl <APIGatewayURL>/sleep/hello
{"message": "Internal server error"}This error will trigger an alarm and also will shown in dashboard. I created a custom metric from CloudWatch Log Group storing the log messages generated from Sleep Lambda function, and then create an alarm and dashboard widget based on this custom metric. Pls refer to cd/cf-sl.yaml#L245 for more details.
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html
Pls go to hands-on#1-4.5 secret API for reference.
We use AWS CodePipeline to pull commits from GitHub, build, test and deploy an API service built on top AWS ECS and Fargate services, in AWS region us-east-1.
The components are
- Two API servers
- Running in Docker containers managed by AWS ECS and Fargate services
- CloudWatch Alarm
- SNS Topic for alarm mail
- CloudWatch dashboard
Where
- Docker container
- is a computer program that performs operating-system-level virtualization.
- AWS ECS (Amazon Elastic Container Service)
- is a highly scalable, high-performance container orchestration service that supports Docker containers
- AWS Fargate
- is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or clusters.
Click following icon to provision AWS CodePipline via AWS CloudFormation (cf-ecs-cp.yaml) in your AWS account.
Pls go to hands-on#1-1. Launch AWS CodePipeline for reference
You can see the CodePipline project starts to trigger a build automatically in few minutes. Due to the CodePipeline will pull repo at first time.
After the service provisioning gets to CREATE_COMPLETED, you can get the endpoint FQDN at the row ALBDNSName in Output tab of AWS CloudFormation console. let's verify your build whether run successfully !
curl http://<ALBDNSName>
Welcome to my homeThe very simple API impl. is api/main.py and running in Docker container (instructions written in a Dockerifle), you can take a look if interested in.
You can see more details in cf-ecs-cp.yaml
Pls go to hands-on#1-3. Take a look at every component in the overall CodePipeline for reference.
The very simple API impl. is api/main.py and running in Docker container (instructions written in a Dockerifle), you can take a look if interested in.
For example APIs pls simply refer to hands-on#1-4. Take a look at our RESTful API service
For deploying new Docker containers to ECS, basically it provides two strategies, 1) Rolling update and 2) Blue/Green deployment. But I still NOT confirm its behaviour yet in this handson, pls do your efforts and share me the answers you got ;)
NOTICE: Pls delete the resources one by one according to the following steps. Otherwise, there would be deletion errors occuring for CloudFormation stacks.
- Approve
ApproveForDeletionaction in CodePipeline
Approve this action will keep to delete CloudFormation stacks
*-ec2and*-ec2-vpcfor hands-on#1*-slfor hands-on#2
- Delete S3 bucket for CodePipeline artifacts
Go to S3 console and delete bucket: *-cicd-test
- Delete ECR repository (optional)
If you are running handson#3, it will build a docker image and push to ECR repository. Pls delete the ECR repository before you go next step.
Go to ECR console and delete bucket: *-repo
- Delete CodePipeline CloudFormation stack
Go to CloudFormation console and delete CodePipeline stack (The stack name was given by you at beginning).
git remote add upstream https://github.com/takeshimiao/aws-cicd.git
git fetch upstream
git checkout master
git merge upstream/master
# you may need to fix the merge conflicts if any
git push origin master