GQL-98: Implements AWS CDK and removes Serverless Framework (#152)

* GQL-98: Implements AWS CDK and removes Serverless Framework

* GQL-98: Updates library

* GQL-98: Fixes python tests

* GQL-98: Adds docker network to sam command, and fixes container weirdness

* GQL-98: Update cdk util library

* GQL-98: Adds the api gateway deployment id to an env variable

* GQL-98: Fixes s3 lambda trigger source

.env.EXAMPLE

@@ -0,0 +1,26 @@
+# Paths to external applications
+CMR_ROOT_URL=https://cmr.earthdata.nasa.gov
+DRAFT_MMT_ROOT_URL=https://draftmmt.earthdata.nasa.gov
+MMT_ROOT_URL=https://mmt.earthdata.nasa.gov
+URS_ROOT_URL=https://urs.earthdata.nasa.gov
+
+# EDL configs, obtain from team member
+# Client ID and Password are used for EDL Groups
+EDL_CLIENT_ID=EDL_CLIENT_ID
+EDL_PASSWORD=EDL_PASSWORD
+
+# JWK and Key ID are used for EDL JWT token verification
+EDL_JWK=EDL_JWK
+EDL_KEY_ID=EDL_KEY_ID
+
+# Graph Database Configurations
+GRAPHDB_HOST=http://localhost
+GRAPHDB_PATH=gremlin
+GRAPHDB_PORT=8182
+
+# Stellate Configrations (https://stellate.co/)
+STELLATE_APP_NAME=""
+STELLATE_KEY=""
+
+# Timeout for lambda
+LAMBDA_TIMEOUT=30

.eslintignore

@@ -0,0 +1 @@
+cdk.out

.github/ISSUE_TEMPLATE/bug_report.md

@@ -7,7 +7,7 @@ assignees: ''
 
 ---
 
-Before contributing to this project, please review [Contribution.md](https://github.com/nasa/cmr-graphql/blob/master/CONTRIBUTING.md).
+Before contributing to this project, please review [Contribution.md](https://github.com/nasa/cmr-graphql/blob/main/CONTRIBUTING.md).
 
 ### Description
 A clear and concise description of what the bug is.

.github/ISSUE_TEMPLATE/custom.md

@@ -7,7 +7,7 @@ assignees: ''
 
 ---
 
-Please review [Contribution.md](https://github.com/nasa/cmr-graphql/blob/master/CONTRIBUTING.md) before contributing to this project.
+Please review [Contribution.md](https://github.com/nasa/cmr-graphql/blob/main/CONTRIBUTING.md) before contributing to this project.
 
 ### Description
 Describe the purpose of the issue

.github/workflows/ci.yml

@@ -80,7 +80,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.9']
+        python-version: ['3.11']
       fail-fast: false
     steps:
       - name: Checkout code
@@ -90,7 +90,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: pip install -r requirements.txt
+        run: pip install -r src/earthdataVarinfo/requirements.txt
       - name: Run tests
         run: python -m unittest discover
         working-directory: ./src/earthdataVarinfo

.gitignore

@@ -27,4 +27,12 @@ junit.xml
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
-*$py.class
+*$py.class
+
+Dockerfile
+.dockerignore
+.env
+.env.prod
+.env.uat
+.env.sit
+.env.local

CONTRIBUTING.md

@@ -0,0 +1,23 @@
+# Contributing to Earthdata Search
+
+Thanks for contributing!
+
+## Making Changes
+
+To allow us to incorporate your changes, please use the following process:
+
+1. Fork this repository to your personal account.
+2. Create a branch and make your changes.
+3. Test the changes locally/in your personal fork.
+4. Submit a pull request to open a discussion about your proposed changes.
+5. The maintainers will talk with you about it and decide to merge or request additional changes.
+
+## Commits
+
+Our ticketing and CI/CD tools are configured to sync statuses amongst each other. Commits play an important role in this process. Please start all commits with the GQL ticket number associated with your feature, task, or bug. All commit messages should follow the format "GQL-XXXX: [Your commit message here]"
+
+## Disclaimer
+
+The Earthdata Search development team will review all pull requests submitted. Only requests that meet the standard of quality set forth by existing code, following the patterns set forth by existing code, and adhering to the design patterns set forth by existing UI elements will be considered and/or accepted.
+
+For general tips on open source contributions, see [Contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/).

README.md

@@ -1,6 +1,5 @@
 # [CMR-GraphQL](https://graphql.earthdata.nasa.gov/api)
 
-[![serverless](http://public.serverless.com/badges/v3.svg)](http://www.serverless.com)
 ![Build Status](https://github.com/nasa/cmr-graphql/workflows/CI/badge.svg?branch=main)
 [![codecov](https://codecov.io/gh/nasa/cmr-graphql/branch/main/graph/badge.svg?token=VZiaLjxD2m)](https://codecov.io/gh/nasa/cmr-graphql)
 
@@ -22,15 +21,36 @@ CMR-GraphQL is an API developed by [NASA](http://nasa.gov) [EOSDIS](https://eart
 
 ## Application Installation
 
-Before running the application you'll want to ensure that all necessary packages are installed by running:
+### Prerequisites
 
-   npm install
+- Docker
+- aws-sam-cli (`brew install aws-sam-cli`)
 
-You will also need Python3.9+ (Ideally installed in a virtual environment) to query the collection generateVariableDrafts field. Run the following to ensure proper operation of this query.
+### Application
 
-    python3 -m venv .venv
-    source .venv/bin/activate
-    pip install -r requirements.txt
+To install Node dependencies:
+
+    npm install
+
+Note: The generateVariableDrafts is currently not working in the development environment. These steps do not work with the CDK changes.
+
+    You will also need Python3.11+ (Ideally installed in a virtual environment) to query the collection generateVariableDrafts field. Run the following to ensure proper operation of this query.
+
+        python3 -m venv .venv
+        source .venv/bin/activate
+        pip install -r requirements.txt
+
+### Environment Variables
+
+We use [dotenvx](https://github.com/dotenvx/dotenvx) to manage multiple `.env` files for various environments. See `.env.EXAMPLE` for an example config, or ask a team member for the full files.
+
+If you want to run any environment locally you will need to create the following files
+
+- .env.prod
+- .env.uat
+- .env.sit
+- .env.local
+  - For running a local copy of CMR
 
 CMR-GraphQL uses a few environment variables for configuring runtime options:
 
@@ -39,16 +59,17 @@ CMR-GraphQL uses a few environment variables for configuring runtime options:
 |CMR_ROOT_URL||URL to ping when retrieving metadata from CMR e.g. https://cmr.earthdata.nasa.gov|
 |MMT_ROOT_URL||URL to ping when retrieving metadata from MMT e.g. https://mmt.earthdata.nasa.gov|
 |DRAFT_MMT_ROOT_URL||URL to ping when retrieving draft metadata from Draft MMT e.g. https://draftmmt.earthdata.nasa.gov|
-|LAMBDA_TIMEOUT|30|Number of seconds to set the Lambda timeout to.|
-|EDL_KEY_ID, EDL_JWK, EDL_CLIENT_ID|For facilitating EDL connection -- obtain these from a dev|
+|EDL_KEY_ID, EDL_JWK, EDL_CLIENT_ID, EDL_PASSWORD||For facilitating EDL connection -- obtain these from a dev|
+
+### Running in developement
 
-### Serverless Framework
+To run locally, with Docker running and aws-sam-cli installed, run
 
-The local development environment for the static assets can be started by executing the command below in the project root directory:
+    npm run start-prod
 
-    serverless offline
+This will run the `cdk synth` command on `cdk/graphql` to build the application locally, and start the AWS SAM CLI pointing to that local build. As you make changes the `cdk synth` command will run and update the running local API.
 
-This will run the application at [http://localhost:3003/dev/api](http://localhost:3003/dev/api)
+This application will be available at [http://127.0.0.1:3013/api](http://127.0.0.1:3013/api)
 
 #### Optional Headers
 
@@ -1873,7 +1894,7 @@ Variables:
 
 #### Local graph database
 
-Normally running GraphQl with `serverless offline` will utilize the `(cmr.earthdata.nasa.gov/graphdb)` endpoint, to query against related collections and duplicate collections in the graph database. To send queries to a locally running graph database, we can use a docker gremlin-server that exposes an HTTP endpoint. This is launched by running
+We use a graph database to query against related collections and duplicate collections. To send queries to a locally running graph database, we can use a docker gremlin-server that exposes an HTTP endpoint. This is launched by running
 
 `docker run -it -p 8182:8182 tinkerpop/gremlin-server conf gremlin-server-rest-modern.yaml`
 

bin/build-python.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cd src/earthdataVarinfo
+pip install -r requirements.txt -t .

bin/deploy-bamboo.sh

@@ -7,60 +7,62 @@ set -eux
 #####################
 
 cat <<EOF > .dockerignore
-node_modules
-.serverless
+**/node_modules
+**/cdk.out
 EOF
 
 cat <<EOF > Dockerfile
 FROM node:22
 COPY . /build
 WORKDIR /build
-RUN npm ci --omit=dev
 RUN apt-get update
 RUN apt-get install -y python3 python3-pip python-is-python3
-RUN pip install -r requirements.txt
+RUN npm ci --omit=dev
+RUN bin/build-python.sh
 EOF
 
 dockerTag=edsc-$bamboo_STAGE_NAME
 docker build -t $dockerTag .
 
 # Convenience function to invoke `docker run` with appropriate env vars instead of baking them into image
 dockerRun() {
-    docker run \
-        -e "AWS_ACCESS_KEY_ID=$bamboo_AWS_ACCESS_KEY_ID" \
-        -e "AWS_SECRET_ACCESS_KEY=$bamboo_AWS_SECRET_ACCESS_KEY" \
-        -e "CLOUDFRONT_BUCKET_NAME=$bamboo_CLOUDFRONT_BUCKET_NAME" \
-        -e "CMR_ROOT_URL=$bamboo_CMR_ROOT_URL" \
-        -e "DMMT_SSL_CERT=$bamboo_DMMT_SSL_CERT" \
-        -e "DRAFT_MMT_ROOT_URL=$bamboo_DRAFT_MMT_ROOT_URL" \
-        -e "EDL_CLIENT_ID=$bamboo_EDL_CLIENT_ID" \
-        -e "EDL_PASSWORD=$bamboo_EDL_PASSWORD" \
-        -e "EDL_JWK=$bamboo_EDL_JWK" \
-        -e "EDL_KEY_ID=$bamboo_EDL_KEY_ID" \
-        -e "GRAPHDB_HOST=$bamboo_GRAPHDB_HOST" \
-        -e "GRAPHDB_PATH=$bamboo_GRAPHDB_PATH" \
-        -e "GRAPHDB_PORT=$bamboo_GRAPHDB_PORT" \
-        -e "LAMBDA_TIMEOUT=$bamboo_LAMBDA_TIMEOUT" \
-        -e "LOG_DESTINATION_ARN=$bamboo_LOG_DESTINATION_ARN" \
-        -e "MMT_ROOT_URL=$bamboo_MMT_ROOT_URL" \
-        -e "NODE_ENV=production" \
-        -e "STELLATE_APP_NAME=$bamboo_STELLATE_APP_NAME" \
-        -e "STELLATE_KEY=$bamboo_STELLATE_KEY" \
-        -e "SUBNET_ID_A=$bamboo_SUBNET_ID_A" \
-        -e "SUBNET_ID_B=$bamboo_SUBNET_ID_B" \
-        -e "VPC_ID=$bamboo_VPC_ID" \
-        $dockerTag "$@"
+  docker run \
+    -e "API_GATEWAY_DEPLOYMENT_ID=$bamboo_API_GATEWAY_DEPLOYMENT_ID" \
+    -e "AWS_ACCESS_KEY_ID=$bamboo_AWS_ACCESS_KEY_ID" \
+    -e "AWS_ACCOUNT=$bamboo_AWS_ACCOUNT" \
+    -e "AWS_SECRET_ACCESS_KEY=$bamboo_AWS_SECRET_ACCESS_KEY" \
+    -e "CLOUDFRONT_BUCKET_NAME=$bamboo_CLOUDFRONT_BUCKET_NAME" \
+    -e "CMR_ROOT_URL=$bamboo_CMR_ROOT_URL" \
+    -e "DMMT_SSL_CERT=$bamboo_DMMT_SSL_CERT" \
+    -e "DOCKER_DEFAULT_PLATFORM=linux/amd64" \
+    -e "DRAFT_MMT_ROOT_URL=$bamboo_DRAFT_MMT_ROOT_URL" \
+    -e "EDL_CLIENT_ID=$bamboo_EDL_CLIENT_ID" \
+    -e "EDL_JWK=$bamboo_EDL_JWK" \
+    -e "EDL_KEY_ID=$bamboo_EDL_KEY_ID" \
+    -e "EDL_PASSWORD=$bamboo_EDL_PASSWORD" \
+    -e "GRAPHDB_HOST=$bamboo_GRAPHDB_HOST" \
+    -e "GRAPHDB_PATH=$bamboo_GRAPHDB_PATH" \
+    -e "GRAPHDB_PORT=$bamboo_GRAPHDB_PORT" \
+    -e "LAMBDA_TIMEOUT=$bamboo_LAMBDA_TIMEOUT" \
+    -e "LOG_DESTINATION_ARN=$bamboo_LOG_DESTINATION_ARN" \
+    -e "MMT_ROOT_URL=$bamboo_MMT_ROOT_URL" \
+    -e "NODE_ENV=production" \
+    -e "STAGE_NAME=$bamboo_STAGE_NAME" \
+    -e "STELLATE_APP_NAME=$bamboo_STELLATE_APP_NAME" \
+    -e "STELLATE_KEY=$bamboo_STELLATE_KEY" \
+    -e "SUBNET_ID_A=$bamboo_SUBNET_ID_A" \
+    -e "SUBNET_ID_B=$bamboo_SUBNET_ID_B" \
+    -e "VPC_ID=$bamboo_VPC_ID" \
+    $dockerTag "$@"
 }
 
-# Execute serverless commands in Docker
+# Execute cdk commands in Docker
 #######################################
 
-stageOpts="--stage $bamboo_STAGE_NAME"
-
 # Deploy AWS Infrastructure Resources
 echo 'Deploying AWS Infrastructure Resources...'
-dockerRun npx serverless deploy $stageOpts --config serverless-infrastructure.yml
+dockerRun npm run deploy-infrastructure
 
 # Deploy AWS Application Resources
 echo 'Deploying AWS Application Resources...'
-dockerRun npx serverless deploy $stageOpts
+dockerRun npm run deploy-application

bin/start-and-watch.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+
+# This will start the watch script and start the sam local api.
+# The watch script watches for code changes and rebuilts the CDK template.
+# The sam local api starts the api locally using the CDK template.
+
+# This is done in a shell script because CTRL-C will kill both processes and shut down any running docker containers.
+(trap 'kill 0' SIGINT; npm run watch & sam local start-api -t ./cdk/graphql/cdk.out/graphql-dev.template.json --warm-containers LAZY --port 3013 --docker-network host)

cdk/graphql-infrastructure/.gitignore

@@ -0,0 +1,8 @@
+*.js
+!jest.config.js
+*.d.ts
+node_modules
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

cdk/graphql-infrastructure/.npmignore

@@ -0,0 +1,6 @@
+*.ts
+!*.d.ts
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

cdk/graphql-infrastructure/README.md

@@ -0,0 +1,14 @@
+# Welcome to your CDK TypeScript project
+
+This is a blank project for CDK development with TypeScript.
+
+The `cdk.json` file tells the CDK Toolkit how to execute your app.
+
+## Useful commands
+
+* `npm run build`   compile typescript to js
+* `npm run watch`   watch for changes and compile
+* `npm run test`    perform the jest unit tests
+* `npx cdk deploy`  deploy this stack to your default AWS account/region
+* `npx cdk diff`    compare deployed stack with current state
+* `npx cdk synth`   emits the synthesized CloudFormation template

cdk/graphql-infrastructure/bin/graphql-infrastructure.ts

@@ -0,0 +1,25 @@
+#!/usr/bin/env node
+import * as cdk from 'aws-cdk-lib';
+import { GraphqlInfrastructureStack } from '../lib/graphql-infrastructure-stack';
+
+const {
+  AWS_ACCOUNT = '1234567890',
+  AWS_REGION = 'us-east-1',
+  STAGE_NAME = 'dev',
+} = process.env;
+const app = new cdk.App();
+new GraphqlInfrastructureStack(app, `graphql-infrastructure-${STAGE_NAME}`, {
+  /* If you don't specify 'env', this stack will be environment-agnostic.
+   * Account/Region-dependent features and context lookups will not work,
+   * but a single synthesized template can be deployed anywhere. */
+
+  /* Uncomment the next line to specialize this stack for the AWS Account
+   * and Region that are implied by the current CLI configuration. */
+  // env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
+
+  /* Uncomment the next line if you know exactly what Account and Region you
+   * want to deploy the stack to. */
+  env: { account: AWS_ACCOUNT, region: AWS_REGION },
+
+  /* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
+});