Skip to content

Home

Jump to bottom
E. Lynette Rayle edited this page Nov 11, 2021 · 11 revisions

Deploying QA Authority Lookup Service to AWS

Overview

We provide templates that help with the process of deploying an Authoritative Data Lookup Service to Amazon Web Services (AWS). The Docker Components will be deployed to various AWS services. Prior to beginning the deploy process, you will need to have basic knowledge of AWS services being used.

Required Background Knowledge

There is an expectation that you have general knowledge of AWS services and an AWS account. Some of the services you will need to be familiar with and set up in advance are:

  • VPC (Virtual Private Cloud) - Within the VPC, you define a public subnet that provides outside access to the application. You will also define a private subnet(s) that provides access to ECS, S3, and EFS components. The setup process will include configurations for things like firewall settings within a security group and ports. (It is common for a VPC to be shared among applications at an institution.)
  • ECR (Elastic Container Registry) - The qa_authority_lookup container is stored in the AWS container registry. It is also available through Docker Hub. TODO: Add links to the containers.
  • ECS (Elastic Container Service) - The application running as a Docker container is deployed through ECS on an EC2 cluster within your VPC. (The templates will provision and set this up.)
  • S3 file storage (Simple Storage Service) - Environment variable configurations are stored in S3. (The templates will provision and set this up.)
  • EFS (Elastic File System) - The database and authority configurations are each stored in EFS which allows them to have mountable drives for access through Docker volumes. (The templates will provision and set these up.)
  • CloudFormation - CloudFormation processes templates and provisions the ECS, S3, and EFS resources based on the configurations in the templates.

Architecture

The architecture follows Docker micro-services paradigm with the application and the database each set up as a service. This is the same whether on AWS or another deployment platform.

This primary deployment components are:

  • Hosted on EC2 through Amazon Elastic Container Service (ECS)
    • The application (app) running within a container.
    • The MySQL database (mysql) running within a container.
  • Stored on S3 (Simple Storage Service)
    • An environment file defining secure environment variables.
  • Mounted on Elastic File System (EFS) or Elastic Block Store (EBS)
    • A volume holding charts generated by the application's performance monitoring module.
    • A volume holding authority configurations that determine which Authoritative Data is available for lookup.
    • A volume holding the database files.

Docker Components on AWS

Running the Templates

The person running the templates must have permissions that allow them to provision the resources created by the templates. Specifically, the user must be able to...

Set values in prerequisites.env

This section describes structures used by the templates that cannot be created or setup by a template. The results of these steps will be recorded as ParameterValues in prerequisites.env. The env file is read by aws-prerequisites.yaml to complete the setup process. For more information on all these steps, see Required Background Knowledge which includes links to official AWS documentation.

Copy prerequisites.env.example and name it prerequisites.env. Edit this file and update parameter values as instructed in the following steps.

AWS VPC ID

Reference: Amazon Virtural Private Cloud - User Guide

  1. Locate your organization's AWS VPC ID and record it as the "ParameterValue" for "ParameterKey": "AWSVpcId" (ex. "vpc-4d4bec2b"). This is required for load balancer configuration and security group rules. NOTE: You may have to create the AWS VPC if this is the first time your organization is using AWS. This is typically setup by your administrator or central IT service.
  2. Create 2 Subnets in your VPC and record the IDs as a comma separated list in the "ParameterValue" for "ParameterKey": "SubnetIds" (ex. "subnet-d7ad4ca1,subnet-83a890da"). These will provide access to ECS, S3, and EFS components.

S3 Bucket

Reference: Amazon Simple Storage Service - User Guide: Working with buckets

  1. Create an S3 bucket and record the bucket name as the "ParameterValue" for "ParameterKey": "S3BucketName" (ex. "lookup-service-bucket")

EFS Mount

Reference: * Amazon Elastic File System - User Guide: Working with Amazon EFS Access Points

  1. Create an EFS database access point and record the TODO as the "ParameterValue" for "ParameterKey": "EFSDatabaseAccessPoint" (ex. "lookup-service-filesystem)

Run aws-prerequisites.yaml template

Set values in parameters.env

Run aws-cloudformation.yaml template

UNDER CONSTRUCTION BEYOND THIS POINT

Upload docker images

Upload the application and database docker images through ECR console

Application docker image

TBD - tentatively - grab from Docker hub - need instructions

The remainder is under construction...

To run the scripts, you must have enough admin privileges to be able to provision all the resources in the template. This deployment assumes that you have an AWS account, and an Identity and Access Management (IAM) account within your AWS account that allows enough access to create, update, and destroy all the necessary resources. If your IAM user lacks sufficient privileges, your deployment will stop with an error message telling you so.

To use this deployment, you will need the following pieces of infrastructure in place in your AWS account before you begin:

  1. A repo containing a successfully built Docker image. This can be an Elastic Container Registry (ECR) repository or a public or private repo hosted elsewhere. Building the application image and pushing it to the repo should already be done, and you will need the full path to the image, including the repository URL and image tag (usually :latest). Full documentation on this process is available at .
  2. A Simple Storage Service (S3) bucket containing an environment file that has been populated with valid values. The file contains secrets and other parameters necessary for the application to run, an .env.example file has been provided as a template. The S3 bucket must contain this file, and the bucket permissions must be configured to allow access from the Amazon ECS task execution IAM role, but it should not be open to the public. Full documentation on setup is available at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/taskdef-envfiles.html.
  3. A filesystem that the containers can mount. At this point, only EFS volumes are supported, EBS will be supported in the future. This deployment assumes one EFS filesystem with two separate access points. The first access point contains the authority files, and it should be pre-populated with the files you want to use. The second will contain the database storage files, and this should provision itself on the first run. While it's not strictly required, you should also have control over a domain space where you are able to create and update DNS records.

Files and Templates

Two templates are provided, a CloudFormation template and a standalone ECS task definition. Both require customization to your environment by populating various values; in the CloudFormation version these values can be passed in as parameters when the stack is created or updated.

Running Templates

Clone this wiki locally