Terragrunt configurations and reusable CI/CD templates for deploying a Kubernetes platform on AWS EKS. Bundles ArgoCD, Istio, Vault, Kong Gateway, security tooling (Kyverno, Falco, Velero), and observability (Loki, Kubecost, compliance scanning) behind a single source of configuration.
This repository contains the environment instantiations (Terragrunt) and bootstrap infrastructure (Terraform). The reusable Terraform modules consumed by the platform layers live in cloudon-one/k8s-platform-modules.
- Architecture
- Repository layout
- Components
- Prerequisites
- Bootstrap and deploy
- Configuration
- CI/CD
- Operations
- Troubleshooting
- Further reading
- Contributing
- License
graph TB
subgraph AWS["AWS Foundation"]
OIDC["GitHub OIDC"]
S3["S3 / DynamoDB / KMS<br/>(Terraform state)"]
Lambda["Secrets Rotation"]
Audit["Security Audit"]
SM["Secrets Manager"]
end
subgraph Core["1. Core Platform"]
Karpenter & ExDNS["External DNS"] & CertMgr["Cert Manager"] & ExtSec["External Secrets"]
end
subgraph Mesh["2. Service Mesh"]
Istio & Kong & Jaeger
end
subgraph Sec["3. Security"]
Kyverno & Falco & Velero
end
subgraph Obs["4. Observability"]
Loki & Kubecost & Compliance["Compliance Scanner"]
end
subgraph Tools["5. Platform Tools"]
ArgoCD & Atlantis & Vault & Airflow
end
OIDC --> S3
Lambda --> SM
SM --> ExtSec
ExtSec --> ArgoCD
ExtSec --> Vault
Layers deploy in numbered order; dependency blocks in each terragrunt.hcl enforce ordering. Destroy in reverse.
.
├── aws-infrastructure/ AWS foundation (Terraform; bootstraps the platform)
│ ├── state-backend/ S3 + DynamoDB + KMS for remote state
│ ├── github-oidc/ GitHub Actions OIDC federation
│ ├── external-secrets-iam/ IRSA roles for External Secrets Operator
│ ├── secrets-rotation-lambda/ Secrets Manager rotation Lambda
│ └── security-audit-automation/ CloudWatch alarms and dashboard
│
├── k8s-platform-tools/ Kubernetes platform (Terragrunt; consumes remote modules)
│ ├── core-platform/ Karpenter, External DNS, Cert Manager, External Secrets
│ ├── service-mesh/ Istio, Kong, Jaeger
│ ├── security/ Kyverno, Falco, Velero
│ ├── observability/ Loki Stack, Kubecost, Compliance Scanner
│ ├── platform-tools/ ArgoCD, Atlantis, Vault, Airflow
│ ├── ci-cd-templates/ Reusable GitHub Actions workflows
│ ├── github-actions-templates/ Per-language test coverage actions
│ ├── common.hcl Shared Terragrunt config (backend, providers, versions)
│ └── platform_vars.yaml Single source of truth for all platform configuration
│
└── .github/workflows/
└── terragrunt-plan-apply-oidc.yaml OIDC-authenticated plan/apply pipeline
The aws-infrastructure/* modules are local Terraform; their terragrunt.hcl files use source = "${get_terragrunt_dir()}" and the .tf lives alongside. The k8s-platform-tools/* components reference remote modules at github.com/cloudon-one/k8s-platform-modules?ref=dev and contain only Terragrunt configuration.
| Layer | Component | Purpose |
|---|---|---|
| Core Platform | Karpenter | Node autoscaling |
| External DNS | DNS record automation | |
| Cert Manager | TLS certificate lifecycle | |
| External Secrets | Sync AWS Secrets Manager → Kubernetes | |
| Service Mesh | Istio | mTLS, traffic management |
| Kong Gateway | North-south API gateway | |
| Jaeger | Distributed tracing | |
| Security | Kyverno | Admission control / policy enforcement |
| Falco | Runtime threat detection (eBPF) | |
| Velero | Backup and disaster recovery | |
| Observability | Loki Stack | Log aggregation |
| Kubecost | Cost monitoring | |
| Compliance Scanner | CIS benchmark scanning | |
| Platform Tools | ArgoCD | GitOps deployment |
| Atlantis | Terraform PR automation | |
| Vault | Secrets management | |
| Airflow | Workflow orchestration | |
| AWS Foundation | State Backend | S3 + DynamoDB + KMS |
| GitHub OIDC | Federated CI/CD authentication | |
| Secrets Rotation Lambda | Automated rotation | |
| Security Audit Automation | CloudWatch monitoring |
Chart and image versions are declared in k8s-platform-tools/platform_vars.yaml. That file is authoritative; this README does not duplicate them.
| Tool | Version | Notes |
|---|---|---|
| Terraform | 1.12.2 |
Pinned in .terraform-version and CI |
| Terragrunt | 1.0.0 |
Pinned in .terragrunt-version and CI |
| AWS CLI | v2 | For authentication and post-deploy verification |
| kubectl | >= 1.28 |
Cluster access after EKS is up |
| Helm | v3 | Required indirectly by remote modules |
An AWS account with permissions to create IAM, S3, DynamoDB, KMS, Lambda, EKS, and VPC resources. Cluster-admin on the resulting EKS cluster is required for the platform layers.
The state backend is bootstrapped with raw Terraform because it cannot store its own state; its terragrunt.hcl declares skip = true. Subsequent components are managed by Terragrunt against that backend.
-
Edit configuration. Set values in
k8s-platform-tools/platform_vars.yaml. At minimum:common.aws_region,common.aws_account_id,common.owner,common.statebucketsuffix,common.vpc_id,common.eks_cluster_name. -
Bootstrap state backend.
cd aws-infrastructure/state-backend terraform init terraform apply -
Bootstrap CI federation and IRSA.
cd ../github-oidc && terragrunt apply cd ../external-secrets-iam && terragrunt apply
-
Deploy platform layers in order.
cd ../../k8s-platform-tools/core-platform && terragrunt run -a -- apply cd ../service-mesh && terragrunt run -a -- apply cd ../security && terragrunt run -a -- apply cd ../observability && terragrunt run -a -- apply cd ../platform-tools && terragrunt run -a -- apply
-
Deploy operational security automation.
cd ../../aws-infrastructure/security-audit-automation && terragrunt apply cd ../secrets-rotation-lambda && terragrunt apply
To destroy, reverse the order. Within k8s-platform-tools/:
terragrunt run -a --working-dir platform-tools -- destroy
terragrunt run -a --working-dir observability -- destroy
terragrunt run -a --working-dir security -- destroy
terragrunt run -a --working-dir service-mesh -- destroy
terragrunt run -a --working-dir core-platform -- destroyPlatform configuration is centralized in k8s-platform-tools/platform_vars.yaml.
| YAML path | Consumed by |
|---|---|
common.* |
All components — region, account, VPC, EKS cluster, OIDC, tags |
Platform.Tools.<name>.inputs |
core-platform/, service-mesh/, platform-tools/ |
Platform.Security.<name>.inputs |
security/ (Kyverno, Falco, Velero) |
Platform.Observability.<name>.inputs |
observability/ (Compliance Scanner) |
The component lookup key is basename(get_terragrunt_dir()). The directory name must match the YAML key exactly.
Environment is selected via the ENV variable (default dev):
ENV=qa terragrunt apply
ENV=prod terragrunt applySensitive values are referenced as AWS Secrets Manager paths:
admin_password: "aws-secretsmanager:///dev/argocd/admin-password"External Secrets Operator (with IRSA) syncs them into the cluster as Kubernetes Secret resources.
.github/workflows/terragrunt-plan-apply-oidc.yaml — manually dispatched, OIDC-authenticated plan and apply pipeline.
- Authentication: GitHub OIDC → AWS STS, no long-lived credentials in CI.
- Inputs:
env(dev/qa/prod),action(plan/apply). - Approval gate:
applyrequires the configured GitHub Environment to approve before running. - Integrity: Terragrunt download is verified against the upstream
SHA256SUMS. - Artifacts: Plan output is uploaded and retained for 30 days.
Under k8s-platform-tools/ci-cd-templates/:
| File | Purpose |
|---|---|
terragrunt-plan-apply.yaml |
Full pipeline with TFSec, Checkov, Infracost, and drift detection |
reusable-docker-build.yaml |
Multi-platform Docker builds with Trivy scanning |
terragrunt-fmt-commit.yaml |
Auto-format with TFLint and open a PR |
get-env-func.yaml |
Map branch name to deployment environment |
terragrunt state list # list resources in the current component
terragrunt state pull > state-backup.tfstate # snapshot before risky changes
terragrunt force-unlock <LOCK_ID> # release a stuck DynamoDB lock# CloudWatch dashboard URL
cd aws-infrastructure/security-audit-automation && terragrunt output dashboard_url
# Compliance scanner report bucket
aws s3 ls "s3://$(yq -r '.common.owner' k8s-platform-tools/platform_vars.yaml)-${ENV:-dev}-compliance-reports/"
# Falco critical events (in-cluster)
kubectl -n falco logs -l app.kubernetes.io/name=falco | grep CRITICAL
# Kyverno policy reports
kubectl get clusterpolicyreport -o yaml
# Velero backup status
velero backup get| Symptom | Resolution |
|---|---|
Error acquiring the state lock |
terragrunt force-unlock <LOCK_ID> — confirm no concurrent run first |
Inputs from platform_vars.yaml not applied |
Directory basename must match the YAML key exactly |
| Remote module fetch fails | Verify Git/SSH access to github.com/cloudon-one/k8s-platform-modules and that ref=dev exists |
Could not assume role in GitHub Actions |
Check OIDC provider thumbprint, role trust policy sub claim, and the requested aud |
| Kyverno blocks a deployment | Inspect kubectl get clusterpolicyreport; if intentional override, set the offending policy's validationFailureAction to Audit |
| ExternalSecret not syncing | kubectl describe externalsecret <name> -n <namespace> — usually IRSA scope or secret path |
Dependency error during apply |
The named dependency hasn't been applied yet; deploy parent layers first |
INFRASTRUCTURE_AS_CODE_SUMMARY.md— IaC design overview and conventions.SECURITY_REVIEW.md— initial security audit findings.SECURITY_IMPLEMENTATION_PLAN.md— phased hardening roadmap.- Phase deployment guides: Phase 1 · Phase 2 · Phase 3.
- Fork and create a feature branch.
- Match existing Terragrunt and Terraform patterns; do not introduce new layout conventions without discussion.
- Configuration changes belong in
k8s-platform-tools/platform_vars.yaml, not in componentterragrunt.hclfiles. - Run
terragrunt hcl format,terraform fmt -recursive, and the scanners inci-cd-templates/terragrunt-plan-apply.yamllocally before opening a PR. - PRs must pass the OIDC plan workflow before merge.
MIT.