Home
We provide templates that help with the process of deploying an Authoritative Data Lookup Service to Amazon Web Services (AWS). The Docker Components will be deployed to various AWS services. Prior to beginning the deploy process, you will need to have basic knowledge of AWS services being used.
There is an expectation that you have general knowledge of AWS services and an AWS account. Some of the services you will need to be familiar with and set up in advance are:
- VPC (Virtual Private Cloud) - Within the VPC, you define a public subnet that provides outside access to the application. You will also define a private subnet(s) that provides access to ECS, S3, and EFS components. The setup process will include configurations for things like firewall settings within a security group and ports. (It is common for a VPC to be shared among applications at an institution.)
- ECR (Elastic Container Registry) - The qa_authority_lookup container is stored in the AWS container registry. It is also available through Docker Hub. TODO: Add links to the containers.
- ECS (Elastic Container Service) - The application running as a Docker container is deployed through ECS on an EC2 cluster within your VPC. (The templates will provision and set this up.)
- S3 file storage (Simple Storage Service) - Environment variable configurations are stored in S3. (The templates will provision and set this up.)
- EFS (Elastic File System) - The database and authority configurations are each stored in EFS which allows them to have mountable drives for access through Docker volumes. (The templates will provision and set these up.)
- CloudFormation - CloudFormation processes templates and provisions the ECS, S3, and EFS resources based on the configurations in the templates.
The architecture follows Docker micro-services paradigm with the application and the database each set up as a service. This is the same whether on AWS or another deployment platform.
This primary deployment components are:
- Hosted on EC2 through Amazon Elastic Container Service (ECS)
- The application (app) running within a container.
- The MySQL database (mysql) running within a container.
- Stored on S3 (Simple Storage Service)
- An environment file defining secure environment variables.
- Mounted on Elastic File System (EFS) or Elastic Block Store (EBS)
- A volume holding charts generated by the application's performance monitoring module.
- A volume holding authority configurations that determine which Authoritative Data is available for lookup.
- A volume holding the database files.
The person running the templates must have permissions that allow them to provision the resources created by the templates. Specifically, the user must be able to...
- create EC2 cluster through ECS (EC2 console)
- request an S3 bucket (S3 console)
- create EFS mounts (EFS console)
- set up IAM access roles (IAM dashboard)
This section describes how to set parameters in prerequisites.env. The env file is read by aws-prerequisites.yaml to complete the setup process for prerequisite resources.
Copy prerequisites.env.example and name it with an .env. extension (e.g. prerequisites.env). Edit this file and update parameter values as instructed in the following steps.
 NOTE: You can have multiple versions of .prerequisites.env to support multi-staged testing of deployments (e.g. .prerequisites-staging.env, .prerequisites-production.env). The name of the file is passed to the template as part of the command to execute the aws-prerequisites.yaml template.
|
Reference: Amazon Virtural Private Cloud - User Guide
You will need a VPC and 2 subnets created in advance. Once you have the IDs, you will set them as the "ParameterValue" for the following keys.
-
"ParameterKey": AWSVpcId- set to the id to the VPC ID (usually provided by your admin) (e.g. "vpc-4d4bec2b") -
"ParameterKey": SubnetIds- set to a comma separated list of the 2 subnet IDs (usually provided by your admin) (e.g. "subnet-d7ad4ca1,subnet-83a890da")
|
Typically, your administrator or central IT service will create these for you. You can request the VPC ID and Subnet IDs to use. See the reference document for more information if your institution does not provide these services.
|
Reference: Amazon Simple Storage Service - User Guide: Working with buckets
The templates will create the S3 bucket. All you need to do is select the name you want to use for the bucket and set it as the "ParameterValue" for the following key.
-
"ParameterKey": S3BucketName- set to the name you want assigned to the S3 bucket (e.g. "lookup-bucket")
|
NOTE: If you are using multi-staged testing of deployments, you will want have a bucket for each deployment (e.g. "lookup-staging-bucket", "lookup-production-bucket"). Set the appropriate value in the env file supporting each deployment.
|
Reference: * Amazon Elastic File System - User Guide: Working with Amazon EFS Access Points
The templates will create the EFS drive and two access points for the database and authority files. All you need to do is to select the names you want to use for these and set them as the "ParameterValue" for the following keys.
-
"ParameterKey": EFSName- set to the name you want assigned to the EFS drive (e.g. "lookup-filesystem") -
"ParameterKey": EFSDatabaseAccessPoint- set to the name you want assigned to the EFS access point for the database (e.g. "lookup-database-ap") -
"ParameterKey": EFSAuthoritiesAccessPoint- set to the name you want assigned to the EFS access point for the authority configuration files (e.g. "lookup-authorities-ap")
|
NOTE: If you are using multi-staged testing of deployments, you will want have an EFS mount for each deployment (e.g. "lookup-staging-filesystem", "lookup-production-filesystem", "lookup-staging-database-ap", "lookup-production-database-ap", "lookup-staging-authorities-ap", "lookup-production-authorities-ap"). Set the appropriate value in the env file supporting each deployment.
|
Reference: AWS cloudformation commands
Reference: AWS cloudformation - create-stack command
The command to create the template that creates the prerequisite resources is...
aws cloudformation create-stack --stack-name lookup-prerequisites \
--template-body file://./aws-prerequisites.yaml \
--parameters file://./prerequisites.env
where,
-
--stack-name- a unique name to hold the created resources. You will get an error if a stack with the name already exists. -
--template-body- the path toaws-prerequisites.yamltemplate -
--parameters- the path toprerequisites.envcreated in the previous section
NOTE: If you used a name other than .prerequisites.env, pass that name as --parameters.
|
|
NOTE: If you are using multi-staged testing of deployments, you will want to name the stack accordingly (e.g. lookup-staging-prerequisites).
|
Reference: AWS cloudformation - describe-stacks command
To check the status of the resources created by this template, execute the following command.
aws cloudformation describe-stacks --stack-name lookup-prerequisites
where,
-
--stack-name- is the name assigned with thecreate-stackcommand
Reference: AWS cloudformation - delete-stack command
During initial testing, you may have the need to recreate the resource stack. Before running the create-stack command again, you will first need to delete the stack.
aws cloudformation destroy-stack --stack-name lookup-prerequisites
where,
-
--stack-name- is the name assigned with thecreate-stackcommand
 WARNING: Running this command cannot be undone. The resources will be destroyed.
|
Upload the application and database docker images through ECR console
TBD - tentatively - grab from Docker hub - need instructions
To run the scripts, you must have enough admin privileges to be able to provision all the resources in the template. This deployment assumes that you have an AWS account, and an Identity and Access Management (IAM) account within your AWS account that allows enough access to create, update, and destroy all the necessary resources. If your IAM user lacks sufficient privileges, your deployment will stop with an error message telling you so.
To use this deployment, you will need the following pieces of infrastructure in place in your AWS account before you begin:
- A repo containing a successfully built Docker image. This can be an Elastic Container Registry (ECR) repository or a public or private repo hosted elsewhere. Building the application image and pushing it to the repo should already be done, and you will need the full path to the image, including the repository URL and image tag (usually :latest). Full documentation on this process is available at .
- A Simple Storage Service (S3) bucket containing an environment file that has been populated with valid values. The file contains secrets and other parameters necessary for the application to run, an .env.example file has been provided as a template. The S3 bucket must contain this file, and the bucket permissions must be configured to allow access from the Amazon ECS task execution IAM role, but it should not be open to the public. Full documentation on setup is available at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/taskdef-envfiles.html.
- A filesystem that the containers can mount. At this point, only EFS volumes are supported, EBS will be supported in the future. This deployment assumes one EFS filesystem with two separate access points. The first access point contains the authority files, and it should be pre-populated with the files you want to use. The second will contain the database storage files, and this should provision itself on the first run. While it's not strictly required, you should also have control over a domain space where you are able to create and update DNS records.
Two templates are provided, a CloudFormation template and a standalone ECS task definition. Both require customization to your environment by populating various values; in the CloudFormation version these values can be passed in as parameters when the stack is created or updated.