|
| 1 | +# AWS S3 to SQS Integration Module |
| 2 | + |
| 3 | +This Terraform module helps you integrate your AWS S3 bucket with CloudQuery Platform and/or ClickHouse by creating the necessary resources and permissions. |
| 4 | + |
| 5 | +## What This Module Creates |
| 6 | + |
| 7 | +- **SQS Queue**: For S3 event notifications when objects are created/updated |
| 8 | +- **S3 Bucket Event Notifications**: Configured to send events to the SQS queue |
| 9 | +- **IAM Role**: With the necessary permissions for CloudQuery Platform and/or ClickHouse |
| 10 | + |
| 11 | +## Prerequisites |
| 12 | + |
| 13 | +- AWS account with permissions to create SQS queues, IAM roles, and modify S3 bucket policies |
| 14 | +- Terraform v0.14+ installed |
| 15 | +- An existing S3 bucket |
| 16 | + |
| 17 | +## Usage |
| 18 | + |
| 19 | +### Basic Setup with CloudQuery Platform Integration |
| 20 | + |
| 21 | +```hcl |
| 22 | +module "s3_to_sqs_integration" { |
| 23 | + source = "github.com/cloudquery/terraform-cloudquery-modules/platform-cx-s3-to-sqs" |
| 24 | +
|
| 25 | + # Existing S3 bucket details |
| 26 | + s3_bucket_id = "my-existing-bucket" |
| 27 | + s3_bucket_arn = "arn:aws:s3:::my-existing-bucket" |
| 28 | +
|
| 29 | + # SQS queue configuration |
| 30 | + queue_name = "my-existing-bucket-notifications" |
| 31 | +
|
| 32 | + # CloudQuery Platform integration |
| 33 | + cloudquery_platform_role_arn = "arn:aws:iam::767397982801:role/cloudquery-platform-production" |
| 34 | + iam_role_name = "cloudquery-platform-access" |
| 35 | + external_id = "cloudquery-${random_id.external_id.hex}" |
| 36 | + require_external_id = true |
| 37 | +} |
| 38 | +
|
| 39 | +resource "random_id" "external_id" { |
| 40 | + byte_length = 8 |
| 41 | +} |
| 42 | +``` |
| 43 | + |
| 44 | +### ClickHouse Integration |
| 45 | + |
| 46 | +```hcl |
| 47 | +module "clickhouse_integration" { |
| 48 | + source = "github.com/cloudquery/terraform-cloudquery-modules/platform-cx-s3-to-sqs" |
| 49 | +
|
| 50 | + # Existing S3 bucket details |
| 51 | + s3_bucket_id = "my-existing-bucket" |
| 52 | + s3_bucket_arn = "arn:aws:s3:::my-existing-bucket" |
| 53 | +
|
| 54 | + # SQS queue configuration (optional for ClickHouse) |
| 55 | + queue_name = "my-existing-bucket-notifications" |
| 56 | +
|
| 57 | + # ClickHouse integration |
| 58 | + clickhouse_role_arn = "arn:aws:iam::012345678901:role/ClickHouse-Integration-Role" |
| 59 | + iam_role_name = "clickhouse-s3-access-role" |
| 60 | +} |
| 61 | +``` |
| 62 | + |
| 63 | +### Dual Integration (CloudQuery Platform and ClickHouse) |
| 64 | + |
| 65 | +```hcl |
| 66 | +module "dual_integration" { |
| 67 | + source = "github.com/cloudquery/terraform-cloudquery-modules/platform-cx-s3-to-sqs" |
| 68 | +
|
| 69 | + # Existing S3 bucket details |
| 70 | + s3_bucket_id = "my-existing-bucket" |
| 71 | + s3_bucket_arn = "arn:aws:s3:::my-existing-bucket" |
| 72 | +
|
| 73 | + # SQS queue configuration |
| 74 | + queue_name = "my-existing-bucket-notifications" |
| 75 | + s3_events = ["s3:ObjectCreated:*"] |
| 76 | + filter_prefix = "data/" # Optional: Filter events by prefix |
| 77 | +
|
| 78 | + # Role configuration for both services |
| 79 | + iam_role_name = "dual-integration-role" |
| 80 | + cloudquery_platform_role_arn = "arn:aws:iam::767397982801:role/cloudquery-platform-production" |
| 81 | + clickhouse_role_arn = "arn:aws:iam::012345678901:role/ClickHouse-Integration-Role" |
| 82 | + external_id = "integration-${random_id.external_id.hex}" |
| 83 | +} |
| 84 | +
|
| 85 | +resource "random_id" "external_id" { |
| 86 | + byte_length = 8 |
| 87 | +} |
| 88 | +``` |
| 89 | + |
| 90 | +## Examples |
| 91 | + |
| 92 | +The module includes several example configurations: |
| 93 | + |
| 94 | +1. [`examples/cloudquery-integration`](examples/cloudquery-integration/main.tf): Setting up CloudQuery Platform integration |
| 95 | +2. [`examples/clickhouse-integration`](examples/clickhouse-integration/main.tf): Setting up ClickHouse integration |
| 96 | +3. [`examples/dual-integration`](examples/dual-integration/main.tf): Setting up both integrations with a single role |
| 97 | + |
| 98 | +## How It Works |
| 99 | + |
| 100 | +1. The module creates an SQS queue in your AWS account |
| 101 | +2. It configures your S3 bucket to send event notifications to the SQS queue |
| 102 | +3. It creates an IAM role that CloudQuery Platform and/or ClickHouse can assume |
| 103 | +4. You provide the Role ARN, External ID (if used), and other details to the service(s) |
| 104 | +5. The services can then securely access your data using these credentials |
| 105 | + |
| 106 | +<!-- BEGIN_TF_DOCS --> |
| 107 | +## Requirements |
| 108 | + |
| 109 | +| Name | Version | |
| 110 | +|------|---------| |
| 111 | +| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 | |
| 112 | +| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.0.0 | |
| 113 | + |
| 114 | +## Providers |
| 115 | + |
| 116 | +| Name | Version | |
| 117 | +|------|---------| |
| 118 | +| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.0.0 | |
| 119 | + |
| 120 | +## Modules |
| 121 | + |
| 122 | +No modules. |
| 123 | + |
| 124 | +## Resources |
| 125 | + |
| 126 | +| Name | Type | |
| 127 | +|------|------| |
| 128 | +| [aws_iam_policy.s3_sqs_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | |
| 129 | +| [aws_iam_policy.s3_to_sqs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource | |
| 130 | +| [aws_iam_role.customer_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | |
| 131 | +| [aws_iam_role_policy_attachment.s3_sqs_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource | |
| 132 | +| [aws_s3_bucket_notification.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification) | resource | |
| 133 | +| [aws_s3_bucket_policy.allow_publish_to_sqs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource | |
| 134 | +| [aws_sqs_queue.dlq](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | |
| 135 | +| [aws_sqs_queue.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | |
| 136 | +| [aws_sqs_queue_redrive_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_redrive_policy) | resource | |
| 137 | +| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source | |
| 138 | +| [aws_iam_policy_document.bucket_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | |
| 139 | +| [aws_iam_policy_document.sqs_consumer](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | |
| 140 | +| [aws_iam_policy_document.sqs_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source | |
| 141 | +| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source | |
| 142 | + |
| 143 | +## Inputs |
| 144 | + |
| 145 | +| Name | Description | Type | Default | Required | |
| 146 | +|------|-------------|------|---------|:--------:| |
| 147 | +| <a name="input_clickhouse_role_arn"></a> [clickhouse\_role\_arn](#input\_clickhouse\_role\_arn) | The ARN of the ClickHouse role that will be allowed to assume the customer role | `string` | `""` | no | |
| 148 | +| <a name="input_cloudquery_platform_role_arn"></a> [cloudquery\_platform\_role\_arn](#input\_cloudquery\_platform\_role\_arn) | The ARN of the CloudQuery Platform role that will be allowed to assume the customer role | `string` | `""` | no | |
| 149 | +| <a name="input_content_based_deduplication"></a> [content\_based\_deduplication](#input\_content\_based\_deduplication) | Enables content-based deduplication for FIFO queues | `bool` | `false` | no | |
| 150 | +| <a name="input_create_consumer_policy"></a> [create\_consumer\_policy](#input\_create\_consumer\_policy) | Whether to create an IAM policy for consumers of this SQS queue | `bool` | `true` | no | |
| 151 | +| <a name="input_deduplication_scope"></a> [deduplication\_scope](#input\_deduplication\_scope) | Specifies whether message deduplication occurs at the message group or queue level | `string` | `"queue"` | no | |
| 152 | +| <a name="input_delay_seconds"></a> [delay\_seconds](#input\_delay\_seconds) | The time in seconds that the delivery of all messages in the queue will be delayed | `number` | `0` | no | |
| 153 | +| <a name="input_dlq_max_receive_count"></a> [dlq\_max\_receive\_count](#input\_dlq\_max\_receive\_count) | The number of times a message can be unsuccessfully dequeued before being moved to the DLQ | `number` | `5` | no | |
| 154 | +| <a name="input_dlq_message_retention_seconds"></a> [dlq\_message\_retention\_seconds](#input\_dlq\_message\_retention\_seconds) | The number of seconds Amazon SQS retains a message in the DLQ | `number` | `1209600` | no | |
| 155 | +| <a name="input_enable_dlq"></a> [enable\_dlq](#input\_enable\_dlq) | Whether to create a dead-letter queue | `bool` | `false` | no | |
| 156 | +| <a name="input_existing_bucket_policy"></a> [existing\_bucket\_policy](#input\_existing\_bucket\_policy) | The existing bucket policy to merge with the S3 notification policy (in JSON format) | `string` | `"{\"Version\":\"2012-10-17\",\"Statement\":[]}"` | no | |
| 157 | +| <a name="input_external_id"></a> [external\_id](#input\_external\_id) | The external ID to use for role assumption (recommended for security) | `string` | `""` | no | |
| 158 | +| <a name="input_fifo_queue"></a> [fifo\_queue](#input\_fifo\_queue) | Boolean designating a FIFO queue | `bool` | `false` | no | |
| 159 | +| <a name="input_fifo_throughput_limit"></a> [fifo\_throughput\_limit](#input\_fifo\_throughput\_limit) | Specifies whether the FIFO queue throughput quota applies to the entire queue or per message group | `string` | `"perQueue"` | no | |
| 160 | +| <a name="input_filter_prefix"></a> [filter\_prefix](#input\_filter\_prefix) | Optional prefix filter for S3 notifications | `string` | `""` | no | |
| 161 | +| <a name="input_filter_suffix"></a> [filter\_suffix](#input\_filter\_suffix) | Optional suffix filter for S3 notifications | `string` | `""` | no | |
| 162 | +| <a name="input_iam_policy_name"></a> [iam\_policy\_name](#input\_iam\_policy\_name) | Name of the IAM policy to create for SQS queue consumers | `string` | `"s3-to-sqs-consumer-policy"` | no | |
| 163 | +| <a name="input_iam_role_name"></a> [iam\_role\_name](#input\_iam\_role\_name) | Name of the IAM role to create for S3 and SQS access | `string` | `"s3-sqs-integration-role"` | no | |
| 164 | +| <a name="input_kms_data_key_reuse_period_seconds"></a> [kms\_data\_key\_reuse\_period\_seconds](#input\_kms\_data\_key\_reuse\_period\_seconds) | The length of time in seconds for which Amazon SQS can reuse a data key to encrypt or decrypt messages | `number` | `300` | no | |
| 165 | +| <a name="input_kms_master_key_id"></a> [kms\_master\_key\_id](#input\_kms\_master\_key\_id) | The ID of an AWS-managed customer master key for Amazon SQS or a custom CMK | `string` | `null` | no | |
| 166 | +| <a name="input_max_message_size"></a> [max\_message\_size](#input\_max\_message\_size) | The limit of how many bytes a message can contain | `number` | `262144` | no | |
| 167 | +| <a name="input_message_retention_seconds"></a> [message\_retention\_seconds](#input\_message\_retention\_seconds) | The number of seconds Amazon SQS retains a message | `number` | `345600` | no | |
| 168 | +| <a name="input_queue_name"></a> [queue\_name](#input\_queue\_name) | Name of the SQS queue to create | `string` | n/a | yes | |
| 169 | +| <a name="input_receive_wait_time_seconds"></a> [receive\_wait\_time\_seconds](#input\_receive\_wait\_time\_seconds) | The time for which a ReceiveMessage call will wait for a message to arrive | `number` | `0` | no | |
| 170 | +| <a name="input_region"></a> [region](#input\_region) | The AWS region to deploy to | `string` | n/a | yes | |
| 171 | +| <a name="input_require_external_id"></a> [require\_external\_id](#input\_require\_external\_id) | Whether to require an external ID when assuming the role | `bool` | `true` | no | |
| 172 | +| <a name="input_s3_bucket_arn"></a> [s3\_bucket\_arn](#input\_s3\_bucket\_arn) | The ARN of the S3 bucket | `string` | n/a | yes | |
| 173 | +| <a name="input_s3_bucket_id"></a> [s3\_bucket\_id](#input\_s3\_bucket\_id) | The ID of the S3 bucket to configure event notifications for | `string` | n/a | yes | |
| 174 | +| <a name="input_s3_events"></a> [s3\_events](#input\_s3\_events) | List of S3 events to trigger notifications for | `list(string)` | <pre>[<br/> "s3:ObjectCreated:*"<br/>]</pre> | no | |
| 175 | +| <a name="input_tags"></a> [tags](#input\_tags) | A map of tags to assign to resources | `map(string)` | `{}` | no | |
| 176 | +| <a name="input_visibility_timeout_seconds"></a> [visibility\_timeout\_seconds](#input\_visibility\_timeout\_seconds) | The visibility timeout for the queue in seconds | `number` | `30` | no | |
| 177 | + |
| 178 | +## Outputs |
| 179 | + |
| 180 | +| Name | Description | |
| 181 | +|------|-------------| |
| 182 | +| <a name="output_bucket_notification_id"></a> [bucket\_notification\_id](#output\_bucket\_notification\_id) | The ID of the S3 bucket notification configuration | |
| 183 | +| <a name="output_customer_role_arn"></a> [customer\_role\_arn](#output\_customer\_role\_arn) | ARN of the IAM role created for CloudQuery/ClickHouse integration | |
| 184 | +| <a name="output_customer_role_name"></a> [customer\_role\_name](#output\_customer\_role\_name) | Name of the IAM role created for CloudQuery/ClickHouse integration | |
| 185 | +| <a name="output_external_id"></a> [external\_id](#output\_external\_id) | External ID used for role assumption (if provided) | |
| 186 | +| <a name="output_iam_policy_arn"></a> [iam\_policy\_arn](#output\_iam\_policy\_arn) | The ARN of the IAM policy for consumers | |
| 187 | +| <a name="output_iam_policy_id"></a> [iam\_policy\_id](#output\_iam\_policy\_id) | The ID of the IAM policy for consumers | |
| 188 | +| <a name="output_iam_policy_name"></a> [iam\_policy\_name](#output\_iam\_policy\_name) | The name of the IAM policy for consumers | |
| 189 | +| <a name="output_sqs_dlq_arn"></a> [sqs\_dlq\_arn](#output\_sqs\_dlq\_arn) | The ARN of the SQS dead-letter queue | |
| 190 | +| <a name="output_sqs_dlq_id"></a> [sqs\_dlq\_id](#output\_sqs\_dlq\_id) | The ID of the SQS dead-letter queue | |
| 191 | +| <a name="output_sqs_dlq_url"></a> [sqs\_dlq\_url](#output\_sqs\_dlq\_url) | The URL of the SQS dead-letter queue | |
| 192 | +| <a name="output_sqs_queue_arn"></a> [sqs\_queue\_arn](#output\_sqs\_queue\_arn) | The ARN of the SQS queue | |
| 193 | +| <a name="output_sqs_queue_id"></a> [sqs\_queue\_id](#output\_sqs\_queue\_id) | The ID of the SQS queue | |
| 194 | +| <a name="output_sqs_queue_url"></a> [sqs\_queue\_url](#output\_sqs\_queue\_url) | The URL of the SQS queue | |
| 195 | +<!-- END_TF_DOCS --> |
| 196 | + |
| 197 | +## Security Best Practices |
| 198 | + |
| 199 | +- Always use an External ID in the trust policy for cross-account access |
| 200 | +- Use a separate IAM role specific to these integrations |
| 201 | +- Consider setting up S3 event notifications with prefix filtering to limit the scope |
| 202 | +- Use a randomly generated External ID rather than a static value |
0 commit comments