This solution deploys AWS infrastructure where security responsibilities are shared between AWS and the customer.
AWS manages:
- Physical infrastructure security and hypervisor layer
- Security of fully managed services (Amazon ECS on AWS Fargate, Amazon SQS, AWS Key Management Service (AWS KMS), Amazon CloudWatch Logs, AWS Secrets Manager)
Customer manages:
- Amazon EC2 guest operating system security and patching
- Amazon EC2 security group configuration and application-level security on EC2 instances
- Replacing placeholder credentials in AWS Secrets Manager with real source storage credentials
- Securing the destination Amazon S3 bucket (Block Public Access, encryption at rest, bucket policies)
- Monitoring Amazon CloudWatch Logs for security events and migration errors
- Reviewing and adjusting IAM policies for their specific security requirements
- Managing AWS KMS key policies and access controls
- Verifying that source endpoint credentials are rotated according to organizational policy
- Deleting credentials from AWS Secrets Manager after migration completes
| Feature | Implementation | Priority |
|---|---|---|
| Encryption at rest (SQS) | Customer-managed AWS KMS keys with automatic annual key material rotation | Critical |
| Encryption at rest (Logs) | Customer-managed AWS KMS keys with automatic annual key material rotation | Critical |
| Encryption at rest (Secrets) | AWS Secrets Manager encrypted with customer-managed AWS KMS key | Critical |
| IAM least privilege | Separate roles for ECS task execution, ECS task operations, and EC2 workers | Critical |
| Network isolation | Security group allows outbound only, no inbound traffic | High |
| Instance metadata security | IMDSv2 enforced (HttpTokens: required) on EC2 instances | High |
| Input validation | Regex allowlists for S3 object keys and bucket names in worker script | High |
| Error handling | try/except blocks around all AWS API calls in lister and worker scripts | High |
| Dead letter queue | Failed messages moved to DLQ after 2 retries for investigation | Medium |
| Scale-in protection | Workers protect themselves from termination during active processing | Medium |
| Custom metrics | Amazon CloudWatch metrics for FilesTransferred, TransferDuration, FailedTransfers | Low |
Note: Automatic key material rotation is enabled on all customer-managed AWS KMS keys. Customers remain responsible for AWS KMS key policy management, access controls, and monitoring key usage.
| Data Type | Classification | Handling |
|---|---|---|
| Source credentials (access key, secret key) | High sensitivity | Stored in AWS Secrets Manager (encrypted at rest with customer-managed AWS KMS key). Replace placeholders immediately post-deployment. Delete after migration completes. |
| Source endpoint URL | Medium sensitivity | Stored in AWS Secrets Manager. |
| Migrated S3 objects | Depends on content | Classification determined by the data being migrated. Verify that destination bucket encryption and access controls match your data classification requirements. |
| SQS messages (object key batches) | Low sensitivity | Contains bucket names and object key paths only. Encrypted with customer-managed AWS KMS key. Automatically deleted after successful processing. |
| Amazon CloudWatch Logs | Low sensitivity | Contains operational logs (file names, transfer status, errors). Encrypted with customer-managed AWS KMS key. 7-day retention. |
This template does not create the destination S3 bucket. Before running a migration, verify that your destination bucket has:
- Block Public Access enabled on all settings
- Default encryption configured (SSE-S3 or SSE-KMS)
- Bucket policy requiring TLS (aws:SecureTransport condition)
- Versioning enabled for data protection
- Server access logging enabled for audit trails
- MFA Delete enabled for buckets containing sensitive or critical data (requires root account credentials)
The template creates three customer-managed AWS KMS keys:
- LogGroupKmsKey — encrypts Amazon CloudWatch Log Groups
- SqsKmsKey — encrypts Amazon SQS queues
- SecretsKmsKey — encrypts AWS Secrets Manager secrets
Key management notes:
- Automatic annual key material rotation is enabled on all keys
- Keys are regional and managed by AWS KMS (no manual backup needed)
- Key policies are version-controlled within the AWS CloudFormation template
- Deleting the CloudFormation stack schedules key deletion with a 30-day waiting period
- During the waiting period, keys can be recovered if needed
- After deletion, encrypted data (logs, queued messages) becomes permanently inaccessible
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Credential exposure | Low | High | Credentials stored in AWS Secrets Manager (encrypted at rest with customer-managed AWS KMS key). No hardcoded credentials in template. |
| Data exfiltration via S3 wildcard permissions | Low | High | Resource: '*' required because destination bucket is user-specified at runtime. Documented as accepted risk with cfn_nag suppression. |
| Unauthorized access to migrated data | Low | High | IAM least-privilege roles. Security group blocks all inbound traffic. |
| Source-side throttling | Medium | Medium | Distributed architecture with multiple workers reduces per-worker request rate. |
| Incomplete migration | Medium | Medium | DLQ captures failed messages. Amazon CloudWatch Logs provide per-file tracking. |
| Supply chain risk (rclone installation) | Low | Medium | rclone installed from official source via HTTPS. Consider pinning a specific version with checksum verification for production use. |
| Cost overrun | Low | Medium | Auto Scaling max set to 5 instances. Scale-to-zero when queue is empty. |
| Network connectivity failure | Medium | Low | Workers automatically retry on failure. Messages return to queue on error. |
| Data residency compliance | Low | High | Verify that source and destination regions comply with your data residency requirements before starting migration. |
This template was scanned with Holmes (cfn-lint, Checkov, cfn-guard, ACAT, Semgrep, Bandit) and cfn_nag. All HIGH severity findings have been addressed with documented compensating controls:
- CKV_AWS_108/CKV_AWS_111 (S3 Resource: '*'): Destination bucket is user-specified at runtime. Source is external S3-compatible storage. Neither bucket ARN is known at deploy time. Documented in DESIGN.md trade-offs table. Suppressed via cfn_nag W11.
- cfnlint/EA012 (External container image): Uses
python:3.13-slimfrom Docker Hub. This is an open-source sample — no custom ECR image is appropriate for a downloadable CloudFormation template. - guru/External3PDownload (curl | bash for rclone): rclone is the core third-party tool. Documented in THIRD-PARTY-LICENSES. Production deployments should pin a specific version with checksum verification.
The template provides application-level logging via Amazon CloudWatch:
/migration/lister— Lister task logs (object enumeration, SQS message sends)/migration/workers— Worker logs (copy operations, errors, metrics)
For comprehensive audit trails, enable the following in your AWS account:
- AWS CloudTrail for API-level logging (Amazon SQS, AWS Secrets Manager, AWS KMS, Amazon S3 operations)
- Amazon S3 server access logging on the destination bucket
- VPC Flow Logs if network-level audit is required (not enabled by default to reduce cost)
If you discover a potential security issue, please notify AWS/Amazon Security via the vulnerability reporting page. Do not create a public GitHub issue.