Cloudformation stack and logging

[BenBlaisdell]

We're looking to move from Cloudwatch docker containers to the awslogs driver for logging in Dart. Currently we have a Cloudformation stack for each cluster which would mean either passing a hardcoded string for the name of the loggroup to each Cloudformation template or using custom resource hackery to reference a loggroup in another stack.

An alternative, and what I think looks like the best option, is to create a single Cloudformation stack for each environment (stage/prod) that has a nested stack for each cluster. This would allow us to include loggroup setup in the base stack and pass the resource into each nested template.

@dannymcpherson, do you remember why you wrote the deployment scripts instead of creating a single stack?

[BenBlaisdell]

@maybeiambatman

[BenBlaisdell]

After further research it looks like you actually can do cross-stack references. I guess the stackoverflow thread I found was outdated. For now I'm going to create a logging stack and use the above technique to pull the log group resources into the various clusters.

Still I think it is worth thinking about using a single cf template to coordinate stacks for each of dart's components rather than using the current deployment script.

[dannymcpherson]

Sorry for the late reply, @BenBlaisdell. Part of the reason for creating separate stacks was to reduce risk and coupling, and increase readability. I still refer to stack outputs in the deployment script.

At the time dart was written, the aws-cloudwatch-log driver was not complete. We have been waiting for it!

Having coordinated stacks sounds simpler, but I think we will always need some sort of deployment script for things like:

• Initial database setup
• Initial ECR population
• Initial engine/subgraph API calls
• Handling for creation/configuration of AWS services that are not yet available in CloudFormation (perhaps more things are supported now, but what about future features?)
• Config template generation
• etc...

The thought definitely crossed my mind about having one CFN template back then, but given the time constraints, I just wasn't able to get something working along those lines. I totally appreciate where you are coming from though. I love the simplicity of the idea.

[BenBlaisdell]

I have the initial structure of what I'm thinking. Right now the new templates are in the /cloudformation/ directory, but that's temporary.

The idea is that there would be one stack from the dart-common template to create ECR repositories and admin groups and a stack from dart-environment for each environment.

Feedback / constructive criticism is always welcome if anyone has strong opinions about this change. Not looking for a full review yet though, as there are still many missing components; I'm still trying to grok the deploy process.

[dannymcpherson]

Thanks, @BenBlaisdell - I'll try to take a look soon.