Skip to content
Merged

Ci #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
1dfe822
initial work, hopefully this exists when merging over the weekend
HoustonBoston Jun 25, 2025
807f5c7
file whitespace cleanup
HoustonBoston Jun 25, 2025
deec723
almost done
HoustonBoston Jun 26, 2025
b3d7435
Made a separate folder for dev and prod stacks. Need to get rid of so…
HoustonBoston Jun 26, 2025
2699b1c
Change into right directory for CI/CD
HoustonBoston Jun 26, 2025
ad16a7d
Can't use run and uses on the same step. Fixed that issue.
HoustonBoston Jun 26, 2025
895f34a
Typo on one of the actions.
HoustonBoston Jun 26, 2025
4687e59
file cleanup
HoustonBoston Jun 26, 2025
73a467c
Added pipeline for master branch
HoustonBoston Jun 27, 2025
7aaaeb3
Update master-branch.yaml
Gurpranked Jun 27, 2025
a9b139e
Update dev-branch.yaml
Gurpranked Jun 27, 2025
66f389b
Added STS Assume Role ARN
HoustonBoston Jun 27, 2025
4624a22
Co-authored-by: Nick Bottari <nbottari9@gmail.com>
HoustonBoston Jun 27, 2025
595614a
Added STS assume role for dev branch pipeline
HoustonBoston Jun 27, 2025
60cd3ab
Changed role-to-assume to secrets.<name>
HoustonBoston Jun 27, 2025
d7e42a2
Added permissions for id-token in the pipeline
HoustonBoston Jun 27, 2025
25f0ecf
Added correct commands to deploy to AWS
HoustonBoston Jun 27, 2025
0b940a0
changed single quote to double quote
HoustonBoston Jun 27, 2025
4f27a34
changed single quote to double quote in all of the formatted strings …
HoustonBoston Jun 27, 2025
a407b92
changed single quote to double quote on line 58
HoustonBoston Jun 27, 2025
5fd03d7
changed single quote to double quote on lines 69 and 79
HoustonBoston Jun 27, 2025
61d7d69
replace os.getenv functions with the environment var name
HoustonBoston Jun 28, 2025
16d0e69
Need to replace all the os.getenv with the variable names at the top.
HoustonBoston Jun 28, 2025
9bb1113
Merge branch 'master' of github.com:UMLCloudComputing/immersion into ci
cjcocokrisp Jun 28, 2025
e6f02a4
Replaced all the os.getenv functions with just the variable names def…
HoustonBoston Jun 30, 2025
1fcdb20
os.exit() -> os._exit()
HoustonBoston Jun 30, 2025
cd027c6
Copied contents of dev CF Stack to prod CF Stack, need to change the …
HoustonBoston Jun 30, 2025
08e2171
File cleanup & will make a PR if CI/CD works for the dev CF stack
HoustonBoston Jun 30, 2025
0df217d
Hopefully this provides the right context for AWS CDK deploy commands
HoustonBoston Jun 30, 2025
e9114f8
Added aws account id as a secret to gh actions
HoustonBoston Jun 30, 2025
1d0a77b
Fixed failure to find Docker image (wrong directory specified)
HoustonBoston Jun 30, 2025
1b7542a
Removed the os._exit function
HoustonBoston Jun 30, 2025
40e8eae
Perhaps this fixes the unable to deserialize error?
HoustonBoston Jun 30, 2025
8971f22
Extracts the string value from the secure string from SSM parameter s…
HoustonBoston Jul 1, 2025
e8d18c4
ecs task definitions have a constant string for the name, hoping that…
HoustonBoston Jul 1, 2025
26a2b6c
aws_cdk not found, so removed that
HoustonBoston Jul 1, 2025
5d6f1e5
Runtime python3.13 not found, changed to python_3_10 as suggested by …
HoustonBoston Jul 1, 2025
dad4200
Fixed directory issues
HoustonBoston Jul 1, 2025
c02baac
Wrong functions were used to retrieve SSM parameter values
HoustonBoston Jul 1, 2025
5bf4f37
Changed to value_from_lookup
HoustonBoston Jul 2, 2025
385483e
Removed .string_value attribute
HoustonBoston Jul 2, 2025
b7d6a1c
Created a CI/CD pipeline for deploying to AWS CDK.
HoustonBoston Jun 30, 2025
c0f0ea6
Merge branch 'ci' of https://github.com/UMLCloudComputing/immersion i…
HoustonBoston Jul 3, 2025
0a90ad9
Fixed indentation
HoustonBoston Jul 3, 2025
6e3994c
Fixed indenting in the prod stack env
HoustonBoston Jul 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/workflows/dev-branch.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: dev-branch-deployment
on:
push:
branches-ignore:
- main
- master
jobs:
dev-deploy-to-aws:
runs-on: ubuntu-latest
env:
CICD_ACCOUNT_ID: ${{ secrets.CICD_ACCOUNT_ID }}

permissions:
id-token: write
contents: read

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
name: Set up python
with:
python-version: 3.11

- name: Install python dependencies
run: pip install -r requirements.txt

- name: Configure AWS profile
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.STS_ASSUME_ROLE_ARN }}
role-session-name: github-actions-cfn-deploy
aws-region: ${{ vars.AWS_REGION }}

- uses: actions/setup-node@v4
name: Install nodejs
with:
node-version: 20

- name: Install CDK CLI with nodejs
run: |
npm install -g aws-cdk

- name: Build the CDK stack using the aws profile
run: |
cd immersion/environments/dev
cdk deploy --app "python3 app.py" --require-approval never
47 changes: 47 additions & 0 deletions .github/workflows/master-branch.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: dev-branch-deployment
on:
pull_request:
branches:
- main
- master
jobs:
prod-deploy-to-aws:
runs-on: ubuntu-latest
env:
CICD_ACCOUNT_ID: ${{ secrets.CICD_ACCOUNT_ID }}

permissions:
id-token: write
contents: read

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
name: Set up python
with:
python-version: 3.11

- name: Install python dependencies
run: pip install -r requirements.txt

- name: Configure AWS profile
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.STS_ASSUME_ROLE_ARN }}
role-session-name: github-actions-cfn-deploy
aws-region: ${{ vars.AWS_REGION }}

- uses: actions/setup-node@v4
name: Install nodejs
with:
node-version: 20

- name: Install CDK CLI with nodejs
run: |
npm install -g aws-cdk

- name: Build the CDK stack using the aws profile
run: |
cd immersion/environments/dev
cdk deploy --app "python3 app.py" --require-approval never
13 changes: 13 additions & 0 deletions immersion/environments/dev/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from immersion_stack import ImmersionStack
import aws_cdk as cdk
import os

app = cdk.App()
ImmersionStack(app, "ImmersionStackDev",
env = {
"account": os.getenv('CICD_ACCOUNT_ID'), # Assuming it's only being deployed with GH Actions
"region": "us-east-1"
}
)

app.synth()
Original file line number Diff line number Diff line change
Expand Up @@ -26,36 +26,56 @@ class ImmersionStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)

if os.getenv('CI') == "true":
APP_NAME = ssm.StringParameter.value_from_lookup(
self,
parameter_name="/immersion/app_name"
)

DISCORD_TOKEN = ssm.StringParameter.value_from_lookup(
self,
parameter_name="/immersion/discord-token-secure"
)
else:
APP_NAME = os.getenv('APP_NAME')
DISCORD_TOKEN = os.getenv('DISCORD_TOKEN')

SSM_PARAMETER_NAME_API = ssm.StringParameter.value_from_lookup(
self,
parameter_name="engage_api_key_test"
)


# DynamoDB Table Definitions
serverTable = dynamodb.TableV2(
self,
f'{os.getenv('APP_NAME')}ServerTable',
f"{APP_NAME}ServerTable",
partition_key=dynamodb.Attribute(name='serverId', type=dynamodb.AttributeType.STRING),
)

onboardingTable = dynamodb.TableV2(
self,
f'{os.getenv('APP_NAME')}OrganizationTable',
f"{APP_NAME}OrganizationTable",
partition_key=dynamodb.Attribute(name='organizationId', type=dynamodb.AttributeType.NUMBER)
)

cacheTable = dynamodb.TableV2(
self,
f'{os.getenv('APP_NAME')}APICacheTable',
f"{APP_NAME}APICacheTable",
partition_key=dynamodb.Attribute(name='clubId', type=dynamodb.AttributeType.STRING),
)

eventTable = dynamodb.TableV2(
self,
f'{os.getenv('APP_NAME')}EventTable',
f"{APP_NAME}EventTable",
partition_key=dynamodb.Attribute(name='eventId', type=dynamodb.AttributeType.STRING),
)

# SQS Queue and Size Metric Defintion
queue = sqs.Queue(
self,
f'{os.getenv('APP_NAME')}DataQueue',
queue_name=f'{os.getenv('APP_NAME')}_Data_Queue',
f"{APP_NAME}DataQueue",
queue_name=f"{APP_NAME}_Data_Queue",
)

scale_metric = queue.metric_approximate_number_of_messages_visible(
Expand All @@ -65,8 +85,8 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:

scale_out_alarm = scale_metric.create_alarm(
self,
f'{os.getenv('APP_NAME')}DataParserScaleOutAlarm',
alarm_name=f'{os.getenv('APP_NAME')}DataProcessScaleOutAlarm',
f"{APP_NAME}DataParserScaleOutAlarm",
alarm_name=f"{APP_NAME}DataProcessScaleOutAlarm",
threshold=1,
evaluation_periods=1,
comparison_operator=cw.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
Expand All @@ -75,8 +95,8 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:

scale_in_alarm = scale_metric.create_alarm(
self,
f'{os.getenv('APP_NAME')}DataParserScaleInAlarm',
alarm_name=f'{os.getenv('APP_NAME')}DataParserScaleInAlarm',
f"{APP_NAME}DataParserScaleInAlarm",
alarm_name=f"{APP_NAME}DataParserScaleInAlarm",
threshold=0,
evaluation_periods=1,
comparison_operator=cw.ComparisonOperator.LESS_THAN_OR_EQUAL_TO_THRESHOLD,
Expand All @@ -92,35 +112,35 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:

cluster = ecs.Cluster(
self,
f'{os.getenv('APP_NAME')}ServiceCluster',
f"{APP_NAME}ServiceCluster",
vpc=vpc
)

# Discord App Container Definition
app_task_defintion = ecs.FargateTaskDefinition(
self,
f'{os.getenv('APP_NAME')}DiscordAppTaskDefinition',
f"{APP_NAME}DiscordAppTaskDefinition",
memory_limit_mib=1024, # 1 GB
cpu=512, # 0.5 vCPU
)

app_task_defintion.add_container(
f'{os.getenv('APP_NAME')}DiscordApp',
f"{APP_NAME}DiscordApp",
image=ecs.ContainerImage.from_docker_image_asset(
DockerImageAsset(
self,
f'{os.getenv('APP_NAME')}DiscordAppDockerImage',
directory='src/discordapp/'
f"{APP_NAME}DiscordAppDockerImage",
directory='../../../src/discordapp/'
)
),
environment={
'DISCORD_TOKEN': os.getenv('DISCORD_TOKEN')
'DISCORD_TOKEN': DISCORD_TOKEN
}
)

app_service = ecs.FargateService(
self,
f'{os.getenv('APP_NAME')}DiscordAppService',
f"{APP_NAME}DiscordAppService",
cluster=cluster,
task_definition=app_task_defintion,
assign_public_ip=True
Expand All @@ -133,19 +153,19 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
# Data Parser Task Definition
parser_task_definition = ecs.FargateTaskDefinition(
self,
f'{os.getenv('APP_NAME')}DataParserTask',
f"{APP_NAME}DataParserTask",
memory_limit_mib=1024, # 1 GB
cpu=512, # 0.5 vCPU
)

parser_task_definition.add_container(
f'{os.getenv('APP_NAME')}DataParser',
f"{APP_NAME}DataParser",
# TODO: REFACTOR TO USE GITHUB CONTAINER REGISTRY!
image=ecs.ContainerImage.from_docker_image_asset(
DockerImageAsset(
self,
f'{os.getenv('APP_NAME')}DataParserImage',
directory='src/data_parser/',
f"{APP_NAME}DataParserImage",
directory='../../../src/data_parser/',
)
),
environment={
Expand All @@ -159,7 +179,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:

parser_service = ecs.FargateService(
self,
f'{os.getenv('APP_NAME')}DataParserService',
f"{APP_NAME}DataParserService",
cluster=cluster,
task_definition=parser_task_definition,
assign_public_ip=True
Expand All @@ -168,7 +188,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
# Scale metrics for data parser service
scaling_target = appautoscaling.ScalableTarget(
self,
id=f'{os.getenv('APP_NAME')}ParserScalingTarget',
id=f"{APP_NAME}ParserScalingTarget",
service_namespace=appautoscaling.ServiceNamespace.ECS,
scalable_dimension='ecs:service:DesiredCount',
min_capacity=0,
Expand Down Expand Up @@ -218,26 +238,27 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
# TODO: figure out how to make a layer for python dependencies
# filter_layer = _lambda.LayerVersion(
# self,
# f'{os.getenv('APP_NAME')}FilterLayer',
# f'{APP_NAME}FilterLayer',

# )

# Data Filter Lambda Functions
engage_api_key_param = ssm.StringParameter.from_secure_string_parameter_attributes(
self,
f'{os.getenv('APP_NAME')}APIKEY',
parameter_name=f'{os.getenv('SSM_PARAMETER_NAME_API')}'
f"{APP_NAME}APIKEY",
parameter_name=f"{os.getenv('SSM_PARAMETER_NAME_API')}"
)

club_information_lambda = lambda_python.PythonFunction(
self,
f'{os.getenv('APP_NAME')}ClubInformationLambda',
runtime=Runtime.PYTHON_3_13,
entry='src/data_filters/onboarding',
f"{APP_NAME}ClubInformationLambda",
runtime=Runtime.PYTHON_3_10,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we change versions?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason it's invalid

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

entry='../../../src/data_filters/onboarding',
handler='lambda_handler',
environment={
'QUEUE_URL': queue.queue_url
},
)

queue.grant_send_messages(club_information_lambda)
engage_api_key_param.grant_read(club_information_lambda)
engage_api_key_param.grant_read(club_information_lambda)
13 changes: 13 additions & 0 deletions immersion/environments/prod/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from immersion_stack import ImmersionStack
import aws_cdk as cdk
import os

app = cdk.App()
ImmersionStack(app, "ImmersionStackDev",
env = {
"account": os.getenv('CICD_ACCOUNT_ID'), # Assuming it's only being deployed with GH Actions
"region": "us-east-1"
}
)

app.synth()
Loading