Terraform, Lambda, and Frontend Changes

.gitignore

@@ -55,5 +55,32 @@ AWS/Repo-information.eml
 # Local data and files
 **/local/
 **/tmp/
+
+# Local .terraform directories
+.terraform/
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Ignore transient lock info files created by terraform apply
+.terraform.tfstate.lock.info
+
+.terraform.lock.hcl
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
 fp-info-cache
-WIBL-backups/
+WIBL-backups/

wibl-python/.gitignore

@@ -165,3 +165,8 @@ local/
 
 # Secrets
 ingest-external*
+
+# Terraform Secrets and configuration
+terraform.tfvars
+default_auth.txt
+backend.hcl

wibl-python/requirements-lambda.txt

@@ -7,3 +7,4 @@ csbschema~=1.1.2
 #   urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with
 #   OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
 urllib3==1.26.18
+deepmerge

wibl-python/scripts/AWSCloudSetup.md

@@ -1,3 +1,7 @@
+# IMPORTANT: 
+This file describes the deprecated scripting build approach. The intended build method is to use Terraform. Follow the 
+instructions in the README.md located in the Terraform folder, which is located here [Terraform_README](./cloud/AWS/Terraform/README.md)
+
 # Packaging and deploying AWS lambdas for DCDB processing
 There are a number of steps required to set up the AWS Lambdas, the S3 buckets, and associated triggers and permissions in order to make processing work in the cloud.  The steps are covered in detail below, and a corresponding set of scripts are available in the `wibl-python/AWS-setup` directory in the repository.  These scripts should *mostly* work, but will likely need some modficiation for a local configuration before being fully executable.  Consider the `configuration-parameters.sh` file first for this.
 

wibl-python/scripts/cloud/AWS/README.md

@@ -1,3 +1,7 @@
+# Important 
+The instructions inside this file describe a deprecated build approach. The intended build method is to use Terraform. 
+Follow the README.md inside the Terraform folder for more information.
+
 # WIBL AWS processing setup instructions and usage
 The following instructions describe how to setup WIBL processing lambdas and WIBL manager
 web service in AWS.

wibl-python/scripts/cloud/AWS/Terraform/README.md

@@ -0,0 +1,133 @@
+# Terraform AWS Build Instructions
+
+## Setup
+The following software items must be installed and configured before continuing
+
+- Docker
+- AWS CLI
+- Terraform
+
+### 1. Changing default configurations
+Inside the [tf-configure-step-1.bash](./tf-configure-step-1.bash) file there is a variable called `PROVIDER_PREFIX`. 
+Replace the value inside the quotation marks with a unique name to apply to all resources. Due to the nature of the cloud, 
+all AWS components must have name unique to its region, type. So, the value given will be used to create unique names
+for all necessary resources. 
+
+Example: `PROVIDER_PREFIX='UNHJHC'`
+
+Once you have updated the value, run the [tf-configure-step-1.bash](./tf-configure-step-1.bash) file. This script will 
+generate a file called `terraform.tfvars` inside the [Terraform](../Terraform) folder. This file has three sections, 
+variables that need to be changed, variables that are optional to change, and variables that can be left alone. 
+
+#### Variables That Need To Be Changed:
+- region: Must be a valid AWS region code. Consider this list for available regions. [AWS_Region_List](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html)
+- DCDB_provider_id: Replace with your given DCDB provider id.
+- superuser_username/password: Choose the main username and password used to access the frontend dashboard.
+- frontend_secret_key: Following the directions is the `terraform.tfvars` file to generate a Django secret. The script 
+to do so is located [here](./generate_secret.sh).
+- origin_secret: This password phrase adds an extra level of security between the CloudFront module and the 
+frontend's load balancer. Run the generate_secret script again to create a new value.
+- DCDB_mode: By default the test mode is active. To switch to using the production URL, set this variable to 1.
+
+#### Variables That Can Optionally Changed: 
+- frontend/manager_db_size: This variable sets the size, in gigabytes, of the respective databases. The frontend database 
+only contains bookkeeping information for user logins and common caching while the manager database contains all the 
+file metadata that is currently in the system. If you come across storage space issues, 99% of the time it will be the 
+manager database. 
+- frontend/manager_username/password: The database usernames and passwords are only used to internally access the database
+contents inside each separate AWS RDS instance. These values can be set for added security.
+
+#### Other Mandatory Replacement Area:
+- In [main.tf](./main.tf) in the section near the top labeled `backend "s3"`, the variables inside must be 
+**manually updated**. They will not follow what is located inside the tfvars file. The variables that need to be updated 
+are `terraform-state-bucket`, `region`, and `terraform-state-key`. They must match their tfvars counter-parts, so copy 
+and pasting their values over is the easier course of action. 
+
+Example 
+```
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 5.0"
+    }
+    docker = {
+      source  = "kreuzwerker/docker"
+      version = "~> 3.0"
+    }
+  }
+
+  backend "s3" {
+    bucket = "REPLACE"
+    region = "REPLACE"
+    key = "REPLACE"
+  }
+
+  required_version = ">= 1.5.0"
+}
+```
+
+- The step 1 script will generate an empty text file name `default_auth.txt`. Inside of this file you must put your DCDB
+provider secret. If the secret is not copied into this file, the final steps of the data pipeline will fail. 
+
+### 2. Bootstrapping Terraform
+Terraform is able to modify and delete computing resources that were created using Terraform. To do this, Terraform must 
+store state information about what resources have been created. This state information can be stored locally on a single 
+compute. However, if the locally stored state information is lost, then you lose the ability to automatically update or 
+delete the computing resources originally created by Terraform. To avoid this problem, Terraform state can be stored 
+remotely in S3. In addition to providing a disaster recovery solution, storing state in S3 allows multiple 
+people/computers to manage a WIBL system created with Terraform, and to do so asynchronously, without having to
+worry about each person's updates overwriting each other.
+
+You can use the script [tf-configure-step-2.bash](tf-configure-step-2.bash) to create an S3 bucket in which to store 
+Terraform state. The name and key for the bucket should have been automatically set in the previous step, but remember 
+to update the [main.tf](./main.tf) so it matches the value in the `terraform.tfvars` file. 
+
+Running tf-configure-step-2.bash will look something like this:
+
+```
+CONTENT_ROOT: /Users/USER_NAME/.../WIBL/wibl-python
+Using AWS_TF_ROOT: /Users/USER_NAME/.../WIBL/wibl-python/scripts/cloud/AWS/Terraform
+Using AWS_PROFILE: default
+Using TF_VARS: /Users/USER_NAME/.../WIBL/wibl-python/scripts/cloud/AWS/Terraform/terraform.tfvars
+Using AWS_REGION: us-east-2
+Using TF_STATE_BUCKET: unhjhc-wibl-tf-state
+Using TF_STATE_KEY: terraform/state/wibl-processing-server-deploy.tfstate
+Using AWS_CLI: aws --profile default --region us-east-2
+Using AWS_ACCOUNT_NUMBER: XXXXXXXXXXXX
+Creating terraform state bucket unhjhc-wibl-tf-state in AWS region us-east-2...
+{
+    "Location": "https://unhjhc-wibl-tf-state.s3.amazonaws.com/"
+}
+Enabling bucket versioning in terraform state bucket unhjhc-wibl-tf-state...
+Done.
+```
+
+### 3. Building The Lambda Package
+This step only works if `docker` is currently running on your system. Inside the outer [AWS](../../AWS) folder is a 
+script called [build-lambda.sh](../build-lambda.sh), run this script. Even if the script is successful, there may still 
+be a lingering `build` or `package` folder, pay these no mind and continue to the next step.
+
+### 4. Building The System With Terraform
+This step also requires `docker` to work, so ensure it is running. First, run the `plan.sh` script inside the Terraform 
+folder. This is where any you will be alerted of any misconfigured or missing variables. If script says "Plan: 125 to 
+add, 0 to change, 0 to destroy." without returning any errors, run the next script `build.sh`. If you do experience any 
+errors when running the `build.sh` script, you need to then run the `destroy.sh` script before you can attempt another 
+build. 
+
+If you experience this error, **Error: Backend configuration changed**, then run the [tf-reconfigure.sh](./tf-reconfigure.sh) 
+script and try to build again.
+
+#### IMPORTANT: The system can take upwards of 15 to 25 minutes to fully build or destroy.
+
+### How To Tear Down The System
+#### WARNING: This script destroys ALL resources, meaning all data put into the system will be unrecoverable.
+To destroy the resources in the cloud, including all AWS and docker resources, run the `destroy.sh` script. 
+The following resources tend to take longer to destroy than their counter-parts. 
+- module.configure-manager-ecs.aws_internet_gateway.ig_public
+- module.configure-manager-ecs.aws_cloudfront_distribution.frontend
+
+This is mostly due to a bug with AWS's resource state management system. AWS believes select lambda functions are still 
+running, which can often leave multiple components hanging on each other waiting for the other to be destroyed. If it 
+seems like this might be the case, using ^c to cancel the script and rerunning it may solve the issue.
+

wibl-python/scripts/cloud/AWS/Terraform/buckets/main.tf

@@ -0,0 +1,59 @@
+data "aws_caller_identity" "current" {}
+data "aws_region" "current" {}
+# TODO: Why is it saying that "bucket_namespace" is not expected here?
+resource "aws_s3_bucket" "incoming_bucket" {
+    bucket = format("%s-%s-%s-an", var.incoming_bucket, data.aws_caller_identity.current.account_id, data.aws_region.current.region)
+    force_destroy = true
+    bucket_namespace = "account-regional"
+}
+
+resource "aws_s3_bucket" "staging_bucket" {
+    bucket = format("%s-%s-%s-an", var.staging_bucket, data.aws_caller_identity.current.account_id, data.aws_region.current.region)
+    force_destroy = true
+    bucket_namespace = "account-regional"
+}
+
+resource "aws_s3_bucket" "viz_bucket" {
+    bucket = format("%s-%s-%s-an", var.viz_bucket, data.aws_caller_identity.current.account_id, data.aws_region.current.region)
+    force_destroy = true
+    bucket_namespace = "account-regional"
+}
+
+resource "aws_s3_bucket" "static_bucket" {
+    bucket = format("%s-%s-%s-an", var.static_bucket, data.aws_caller_identity.current.account_id, data.aws_region.current.region)
+    force_destroy = true
+    bucket_namespace = "account-regional"
+}
+
+data "aws_iam_policy_document" "static_bucket_policy" {
+  statement {
+    actions = ["s3:GetObject"]
+
+    resources = [
+      "${aws_s3_bucket.static_bucket.arn}/static/*"
+    ]
+
+    principals {
+      type        = "AWS"
+      identifiers = [var.oai_iam_arn]
+    }
+  }
+}
+
+resource "aws_s3_bucket_policy" "static" {
+  bucket = aws_s3_bucket.static_bucket.id
+  policy = data.aws_iam_policy_document.static_bucket_policy.json
+}
+
+resource "aws_s3_bucket_cors_configuration" "static_files" {
+  bucket = aws_s3_bucket.static_bucket.id
+
+  cors_rule {
+    allowed_methods = ["GET", "HEAD"]
+    allowed_origins = [var.alb_url]
+    allowed_headers = ["*"]
+    max_age_seconds = 3000
+  }
+  depends_on = [aws_s3_bucket.static_bucket]
+}
+

wibl-python/scripts/cloud/AWS/Terraform/buckets/outputs.tf

@@ -0,0 +1,34 @@
+output "incoming_bucket_arn" {
+    value = aws_s3_bucket.incoming_bucket.arn
+}
+
+output "incoming_bucket_id" {
+    value = aws_s3_bucket.incoming_bucket.id
+}
+output "staging_bucket_arn"  {
+    value = aws_s3_bucket.staging_bucket.arn
+}
+
+output "viz_bucket_arn" {
+    value = aws_s3_bucket.viz_bucket.arn
+}
+
+output "static_bucket_regional_dns_name" {
+    value = aws_s3_bucket.static_bucket.bucket_regional_domain_name
+}
+
+output "incoming_bucket_full_name" {
+    value = aws_s3_bucket.incoming_bucket.bucket
+}
+
+output "staging_bucket_full_name" {
+    value = aws_s3_bucket.staging_bucket.bucket
+}
+
+output "static_bucket_full_name" {
+    value = aws_s3_bucket.static_bucket.bucket
+}
+
+output "viz_bucket_full_name" {
+    value = aws_s3_bucket.viz_bucket.bucket
+}

wibl-python/scripts/cloud/AWS/Terraform/buckets/variables.tf

@@ -0,0 +1,28 @@
+variable "incoming_bucket" {
+    description = "Name of the incoming s3 bucket"
+    type = string
+}
+
+variable "staging_bucket" {
+    description = "Name of the staging s3 bucket"
+    type = string
+}
+
+variable "viz_bucket" {
+    description = "Name of the viz s3 bucket"
+    type = string
+}
+
+variable "static_bucket" {
+    description = "Name of the static bucket"
+    type = string
+}
+
+variable "alb_url" {
+    description = "Frontend cloudfront distribution url"
+    type = string
+}
+
+variable "oai_iam_arn" {
+    type = string
+}

wibl-python/scripts/cloud/AWS/Terraform/build.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+WIBL_BUILD_LOCATION=$(git rev-parse --show-toplevel)/wibl-python/awsbuild
+ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
+SRC_PATH=$(git rev-parse --show-toplevel)/wibl-python
+TF_LOG=DEBUG
+terraform init -backend-config=backend.hcl && terraform apply -auto-approve -var="src_path=${SRC_PATH}" -var="wibl_build_path=${WIBL_BUILD_LOCATION}" -var="account_number=${ACCOUNT_NUMBER}"

wibl-python/scripts/cloud/AWS/Terraform/destroy.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+WIBL_BUILD_LOCATION=$(git rev-parse --show-toplevel)/wibl-python/awsbuild
+ACCOUNT_NUMBER=$(aws sts get-caller-identity --query Account --output text)
+SRC_PATH=$(git rev-parse --show-toplevel)/wibl-python
+terraform destroy -auto-approve -var="wibl_build_path=${WIBL_BUILD_LOCATION}" -var="src_path=${SRC_PATH}" -var="account_number=${ACCOUNT_NUMBER}"

wibl-python/scripts/cloud/AWS/Terraform/generate_secret.sh

@@ -0,0 +1 @@
+dd if=/dev/random bs=4096 count=1 status=none | sha512sum --quiet