Skip to content
This repository was archived by the owner on Mar 30, 2026. It is now read-only.

Commit 8b0a3ab

Browse files
feat: Add platform-cx-s3-to-sqs module (#39)
* feat: Add platform-cx-s3-to-sqs module * chore: move provider config to separate file * chore: ci fixes * chore: update providers, add versions file * docs: update terraform docs * chore: update examples files * chore: update paths for examples * chore: fix paths * chore: update outputs in examples * chore: small updates * docs: update terraform docs * chore: add region to examples * chore: update outputs * chore: small changes for tflint error * docs: update terraform docs --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 6d435bd commit 8b0a3ab

17 files changed

Lines changed: 1643 additions & 0 deletions

File tree

platform-cx-s3-to-sqs/README.md

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# AWS S3 to SQS Integration Module
2+
3+
This Terraform module helps you integrate your AWS S3 bucket with CloudQuery Platform and/or ClickHouse by creating the necessary resources and permissions.
4+
5+
## What This Module Creates
6+
7+
- **SQS Queue**: For S3 event notifications when objects are created/updated
8+
- **S3 Bucket Event Notifications**: Configured to send events to the SQS queue
9+
- **IAM Role**: With the necessary permissions for CloudQuery Platform and/or ClickHouse
10+
11+
## Prerequisites
12+
13+
- AWS account with permissions to create SQS queues, IAM roles, and modify S3 bucket policies
14+
- Terraform v0.14+ installed
15+
- An existing S3 bucket
16+
17+
## Usage
18+
19+
### Basic Setup with CloudQuery Platform Integration
20+
21+
```hcl
22+
module "s3_to_sqs_integration" {
23+
source = "github.com/cloudquery/terraform-cloudquery-modules/platform-cx-s3-to-sqs"
24+
25+
# Existing S3 bucket details
26+
s3_bucket_id = "my-existing-bucket"
27+
s3_bucket_arn = "arn:aws:s3:::my-existing-bucket"
28+
29+
# SQS queue configuration
30+
queue_name = "my-existing-bucket-notifications"
31+
32+
# CloudQuery Platform integration
33+
cloudquery_platform_role_arn = "arn:aws:iam::767397982801:role/cloudquery-platform-production"
34+
iam_role_name = "cloudquery-platform-access"
35+
external_id = "cloudquery-${random_id.external_id.hex}"
36+
require_external_id = true
37+
}
38+
39+
resource "random_id" "external_id" {
40+
byte_length = 8
41+
}
42+
```
43+
44+
### ClickHouse Integration
45+
46+
```hcl
47+
module "clickhouse_integration" {
48+
source = "github.com/cloudquery/terraform-cloudquery-modules/platform-cx-s3-to-sqs"
49+
50+
# Existing S3 bucket details
51+
s3_bucket_id = "my-existing-bucket"
52+
s3_bucket_arn = "arn:aws:s3:::my-existing-bucket"
53+
54+
# SQS queue configuration (optional for ClickHouse)
55+
queue_name = "my-existing-bucket-notifications"
56+
57+
# ClickHouse integration
58+
clickhouse_role_arn = "arn:aws:iam::012345678901:role/ClickHouse-Integration-Role"
59+
iam_role_name = "clickhouse-s3-access-role"
60+
}
61+
```
62+
63+
### Dual Integration (CloudQuery Platform and ClickHouse)
64+
65+
```hcl
66+
module "dual_integration" {
67+
source = "github.com/cloudquery/terraform-cloudquery-modules/platform-cx-s3-to-sqs"
68+
69+
# Existing S3 bucket details
70+
s3_bucket_id = "my-existing-bucket"
71+
s3_bucket_arn = "arn:aws:s3:::my-existing-bucket"
72+
73+
# SQS queue configuration
74+
queue_name = "my-existing-bucket-notifications"
75+
s3_events = ["s3:ObjectCreated:*"]
76+
filter_prefix = "data/" # Optional: Filter events by prefix
77+
78+
# Role configuration for both services
79+
iam_role_name = "dual-integration-role"
80+
cloudquery_platform_role_arn = "arn:aws:iam::767397982801:role/cloudquery-platform-production"
81+
clickhouse_role_arn = "arn:aws:iam::012345678901:role/ClickHouse-Integration-Role"
82+
external_id = "integration-${random_id.external_id.hex}"
83+
}
84+
85+
resource "random_id" "external_id" {
86+
byte_length = 8
87+
}
88+
```
89+
90+
## Examples
91+
92+
The module includes several example configurations:
93+
94+
1. [`examples/cloudquery-integration`](examples/cloudquery-integration/main.tf): Setting up CloudQuery Platform integration
95+
2. [`examples/clickhouse-integration`](examples/clickhouse-integration/main.tf): Setting up ClickHouse integration
96+
3. [`examples/dual-integration`](examples/dual-integration/main.tf): Setting up both integrations with a single role
97+
98+
## How It Works
99+
100+
1. The module creates an SQS queue in your AWS account
101+
2. It configures your S3 bucket to send event notifications to the SQS queue
102+
3. It creates an IAM role that CloudQuery Platform and/or ClickHouse can assume
103+
4. You provide the Role ARN, External ID (if used), and other details to the service(s)
104+
5. The services can then securely access your data using these credentials
105+
106+
<!-- BEGIN_TF_DOCS -->
107+
## Requirements
108+
109+
| Name | Version |
110+
|------|---------|
111+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
112+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.0.0 |
113+
114+
## Providers
115+
116+
| Name | Version |
117+
|------|---------|
118+
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.0.0 |
119+
120+
## Modules
121+
122+
No modules.
123+
124+
## Resources
125+
126+
| Name | Type |
127+
|------|------|
128+
| [aws_iam_policy.s3_sqs_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
129+
| [aws_iam_policy.s3_to_sqs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
130+
| [aws_iam_role.customer_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
131+
| [aws_iam_role_policy_attachment.s3_sqs_access](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
132+
| [aws_s3_bucket_notification.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification) | resource |
133+
| [aws_s3_bucket_policy.allow_publish_to_sqs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_policy) | resource |
134+
| [aws_sqs_queue.dlq](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource |
135+
| [aws_sqs_queue.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource |
136+
| [aws_sqs_queue_redrive_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_redrive_policy) | resource |
137+
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
138+
| [aws_iam_policy_document.bucket_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
139+
| [aws_iam_policy_document.sqs_consumer](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
140+
| [aws_iam_policy_document.sqs_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
141+
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
142+
143+
## Inputs
144+
145+
| Name | Description | Type | Default | Required |
146+
|------|-------------|------|---------|:--------:|
147+
| <a name="input_clickhouse_role_arn"></a> [clickhouse\_role\_arn](#input\_clickhouse\_role\_arn) | The ARN of the ClickHouse role that will be allowed to assume the customer role | `string` | `""` | no |
148+
| <a name="input_cloudquery_platform_role_arn"></a> [cloudquery\_platform\_role\_arn](#input\_cloudquery\_platform\_role\_arn) | The ARN of the CloudQuery Platform role that will be allowed to assume the customer role | `string` | `""` | no |
149+
| <a name="input_content_based_deduplication"></a> [content\_based\_deduplication](#input\_content\_based\_deduplication) | Enables content-based deduplication for FIFO queues | `bool` | `false` | no |
150+
| <a name="input_create_consumer_policy"></a> [create\_consumer\_policy](#input\_create\_consumer\_policy) | Whether to create an IAM policy for consumers of this SQS queue | `bool` | `true` | no |
151+
| <a name="input_deduplication_scope"></a> [deduplication\_scope](#input\_deduplication\_scope) | Specifies whether message deduplication occurs at the message group or queue level | `string` | `"queue"` | no |
152+
| <a name="input_delay_seconds"></a> [delay\_seconds](#input\_delay\_seconds) | The time in seconds that the delivery of all messages in the queue will be delayed | `number` | `0` | no |
153+
| <a name="input_dlq_max_receive_count"></a> [dlq\_max\_receive\_count](#input\_dlq\_max\_receive\_count) | The number of times a message can be unsuccessfully dequeued before being moved to the DLQ | `number` | `5` | no |
154+
| <a name="input_dlq_message_retention_seconds"></a> [dlq\_message\_retention\_seconds](#input\_dlq\_message\_retention\_seconds) | The number of seconds Amazon SQS retains a message in the DLQ | `number` | `1209600` | no |
155+
| <a name="input_enable_dlq"></a> [enable\_dlq](#input\_enable\_dlq) | Whether to create a dead-letter queue | `bool` | `false` | no |
156+
| <a name="input_existing_bucket_policy"></a> [existing\_bucket\_policy](#input\_existing\_bucket\_policy) | The existing bucket policy to merge with the S3 notification policy (in JSON format) | `string` | `"{\"Version\":\"2012-10-17\",\"Statement\":[]}"` | no |
157+
| <a name="input_external_id"></a> [external\_id](#input\_external\_id) | The external ID to use for role assumption (recommended for security) | `string` | `""` | no |
158+
| <a name="input_fifo_queue"></a> [fifo\_queue](#input\_fifo\_queue) | Boolean designating a FIFO queue | `bool` | `false` | no |
159+
| <a name="input_fifo_throughput_limit"></a> [fifo\_throughput\_limit](#input\_fifo\_throughput\_limit) | Specifies whether the FIFO queue throughput quota applies to the entire queue or per message group | `string` | `"perQueue"` | no |
160+
| <a name="input_filter_prefix"></a> [filter\_prefix](#input\_filter\_prefix) | Optional prefix filter for S3 notifications | `string` | `""` | no |
161+
| <a name="input_filter_suffix"></a> [filter\_suffix](#input\_filter\_suffix) | Optional suffix filter for S3 notifications | `string` | `""` | no |
162+
| <a name="input_iam_policy_name"></a> [iam\_policy\_name](#input\_iam\_policy\_name) | Name of the IAM policy to create for SQS queue consumers | `string` | `"s3-to-sqs-consumer-policy"` | no |
163+
| <a name="input_iam_role_name"></a> [iam\_role\_name](#input\_iam\_role\_name) | Name of the IAM role to create for S3 and SQS access | `string` | `"s3-sqs-integration-role"` | no |
164+
| <a name="input_kms_data_key_reuse_period_seconds"></a> [kms\_data\_key\_reuse\_period\_seconds](#input\_kms\_data\_key\_reuse\_period\_seconds) | The length of time in seconds for which Amazon SQS can reuse a data key to encrypt or decrypt messages | `number` | `300` | no |
165+
| <a name="input_kms_master_key_id"></a> [kms\_master\_key\_id](#input\_kms\_master\_key\_id) | The ID of an AWS-managed customer master key for Amazon SQS or a custom CMK | `string` | `null` | no |
166+
| <a name="input_max_message_size"></a> [max\_message\_size](#input\_max\_message\_size) | The limit of how many bytes a message can contain | `number` | `262144` | no |
167+
| <a name="input_message_retention_seconds"></a> [message\_retention\_seconds](#input\_message\_retention\_seconds) | The number of seconds Amazon SQS retains a message | `number` | `345600` | no |
168+
| <a name="input_queue_name"></a> [queue\_name](#input\_queue\_name) | Name of the SQS queue to create | `string` | n/a | yes |
169+
| <a name="input_receive_wait_time_seconds"></a> [receive\_wait\_time\_seconds](#input\_receive\_wait\_time\_seconds) | The time for which a ReceiveMessage call will wait for a message to arrive | `number` | `0` | no |
170+
| <a name="input_region"></a> [region](#input\_region) | The AWS region to deploy to | `string` | n/a | yes |
171+
| <a name="input_require_external_id"></a> [require\_external\_id](#input\_require\_external\_id) | Whether to require an external ID when assuming the role | `bool` | `true` | no |
172+
| <a name="input_s3_bucket_arn"></a> [s3\_bucket\_arn](#input\_s3\_bucket\_arn) | The ARN of the S3 bucket | `string` | n/a | yes |
173+
| <a name="input_s3_bucket_id"></a> [s3\_bucket\_id](#input\_s3\_bucket\_id) | The ID of the S3 bucket to configure event notifications for | `string` | n/a | yes |
174+
| <a name="input_s3_events"></a> [s3\_events](#input\_s3\_events) | List of S3 events to trigger notifications for | `list(string)` | <pre>[<br/> "s3:ObjectCreated:*"<br/>]</pre> | no |
175+
| <a name="input_tags"></a> [tags](#input\_tags) | A map of tags to assign to resources | `map(string)` | `{}` | no |
176+
| <a name="input_visibility_timeout_seconds"></a> [visibility\_timeout\_seconds](#input\_visibility\_timeout\_seconds) | The visibility timeout for the queue in seconds | `number` | `30` | no |
177+
178+
## Outputs
179+
180+
| Name | Description |
181+
|------|-------------|
182+
| <a name="output_bucket_notification_id"></a> [bucket\_notification\_id](#output\_bucket\_notification\_id) | The ID of the S3 bucket notification configuration |
183+
| <a name="output_customer_role_arn"></a> [customer\_role\_arn](#output\_customer\_role\_arn) | ARN of the IAM role created for CloudQuery/ClickHouse integration |
184+
| <a name="output_customer_role_name"></a> [customer\_role\_name](#output\_customer\_role\_name) | Name of the IAM role created for CloudQuery/ClickHouse integration |
185+
| <a name="output_external_id"></a> [external\_id](#output\_external\_id) | External ID used for role assumption (if provided) |
186+
| <a name="output_iam_policy_arn"></a> [iam\_policy\_arn](#output\_iam\_policy\_arn) | The ARN of the IAM policy for consumers |
187+
| <a name="output_iam_policy_id"></a> [iam\_policy\_id](#output\_iam\_policy\_id) | The ID of the IAM policy for consumers |
188+
| <a name="output_iam_policy_name"></a> [iam\_policy\_name](#output\_iam\_policy\_name) | The name of the IAM policy for consumers |
189+
| <a name="output_sqs_dlq_arn"></a> [sqs\_dlq\_arn](#output\_sqs\_dlq\_arn) | The ARN of the SQS dead-letter queue |
190+
| <a name="output_sqs_dlq_id"></a> [sqs\_dlq\_id](#output\_sqs\_dlq\_id) | The ID of the SQS dead-letter queue |
191+
| <a name="output_sqs_dlq_url"></a> [sqs\_dlq\_url](#output\_sqs\_dlq\_url) | The URL of the SQS dead-letter queue |
192+
| <a name="output_sqs_queue_arn"></a> [sqs\_queue\_arn](#output\_sqs\_queue\_arn) | The ARN of the SQS queue |
193+
| <a name="output_sqs_queue_id"></a> [sqs\_queue\_id](#output\_sqs\_queue\_id) | The ID of the SQS queue |
194+
| <a name="output_sqs_queue_url"></a> [sqs\_queue\_url](#output\_sqs\_queue\_url) | The URL of the SQS queue |
195+
<!-- END_TF_DOCS -->
196+
197+
## Security Best Practices
198+
199+
- Always use an External ID in the trust policy for cross-account access
200+
- Use a separate IAM role specific to these integrations
201+
- Consider setting up S3 event notifications with prefix filtering to limit the scope
202+
- Use a randomly generated External ID rather than a static value
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
<!-- BEGIN_TF_DOCS -->
2+
## Requirements
3+
4+
No requirements.
5+
6+
## Providers
7+
8+
| Name | Version |
9+
|------|---------|
10+
| <a name="provider_random"></a> [random](#provider\_random) | n/a |
11+
12+
## Modules
13+
14+
| Name | Source | Version |
15+
|------|--------|---------|
16+
| <a name="module_s3_to_sqs_integration"></a> [s3\_to\_sqs\_integration](#module\_s3\_to\_sqs\_integration) | github.com/cloudquery/terraform-aws-s3-to-sqs | n/a |
17+
18+
## Resources
19+
20+
| Name | Type |
21+
|------|------|
22+
| [random_id.external_id](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
23+
24+
## Inputs
25+
26+
No inputs.
27+
28+
## Outputs
29+
30+
| Name | Description |
31+
|------|-------------|
32+
| <a name="output_configuration_instructions"></a> [configuration\_instructions](#output\_configuration\_instructions) | Instructions for providing the role information to ClickHouse Cloud |
33+
| <a name="output_role_arn"></a> [role\_arn](#output\_role\_arn) | The ARN of the IAM role to provide to ClickHouse Cloud |
34+
| <a name="output_role_external_id"></a> [role\_external\_id](#output\_role\_external\_id) | The external ID to provide to ClickHouse Cloud (keep this secure) |
35+
| <a name="output_sqs_queue_arn"></a> [sqs\_queue\_arn](#output\_sqs\_queue\_arn) | The ARN of the SQS queue |
36+
| <a name="output_sqs_queue_url"></a> [sqs\_queue\_url](#output\_sqs\_queue\_url) | The URL of the SQS queue receiving S3 event notifications |
37+
<!-- END_TF_DOCS -->
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Example for ClickHouse integration with existing S3 bucket
2+
# This creates an IAM role that allows ClickHouse to access your S3 data
3+
4+
locals {
5+
# ClickHouse IAM role ARN - THIS SHOULD BE THE ACTUAL ROLE ARN PROVIDED BY CLICKHOUSE
6+
clickhouse_role_arn = "arn:aws:iam::191110999071:role/CH-S3-steel-mv-95-ue1-42-Role"
7+
8+
# Your existing S3 bucket information
9+
existing_bucket_name = "your-existing-bucket"
10+
existing_bucket_arn = "arn:aws:s3:::your-existing-bucket"
11+
}
12+
13+
module "clickhouse_integration" {
14+
source = "../../"
15+
16+
# Region where the resources will be created
17+
region = "us-west-2"
18+
19+
# Existing S3 bucket details
20+
s3_bucket_id = local.existing_bucket_name
21+
s3_bucket_arn = local.existing_bucket_arn
22+
23+
# The SQS queue is optional for ClickHouse, but we'll create it anyway
24+
# to maintain compatibility with the module
25+
queue_name = "${local.existing_bucket_name}-notifications"
26+
27+
# Create IAM role with appropriate trust policy for ClickHouse
28+
iam_role_name = "clickhouse-s3-access-role"
29+
clickhouse_role_arn = local.clickhouse_role_arn
30+
}
31+
32+
# Output the information needed to provide to ClickHouse
33+
output "role_arn" {
34+
description = "The ARN of the IAM role to provide to ClickHouse"
35+
value = module.clickhouse_integration.customer_role_arn
36+
}
37+
38+
output "bucket_name" {
39+
description = "The name of the S3 bucket to provide to ClickHouse"
40+
value = local.existing_bucket_name
41+
}
42+
43+
output "configuration_instructions" {
44+
description = "Instructions for ClickHouse integration"
45+
value = <<-EOT
46+
## ClickHouse Integration Instructions
47+
48+
Provide the following information to ClickHouse:
49+
50+
1. Role ARN: ${module.clickhouse_integration.customer_role_arn}
51+
2. S3 Bucket Name: ${local.existing_bucket_name}
52+
53+
This role has been configured with the exact permissions required by ClickHouse
54+
to access your S3 bucket data.
55+
EOT
56+
}
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
<!-- BEGIN_TF_DOCS -->
2+
## Requirements
3+
4+
No requirements.
5+
6+
## Providers
7+
8+
| Name | Version |
9+
|------|---------|
10+
| <a name="provider_aws"></a> [aws](#provider\_aws) | n/a |
11+
| <a name="provider_random"></a> [random](#provider\_random) | n/a |
12+
13+
## Modules
14+
15+
| Name | Source | Version |
16+
|------|--------|---------|
17+
| <a name="module_s3_to_sqs_integration"></a> [s3\_to\_sqs\_integration](#module\_s3\_to\_sqs\_integration) | github.com/cloudquery/terraform-aws-s3-to-sqs | n/a |
18+
19+
## Resources
20+
21+
| Name | Type |
22+
|------|------|
23+
| [aws_s3_bucket.data_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
24+
| [aws_s3_bucket_acl.data_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_acl) | resource |
25+
| [aws_s3_bucket_ownership_controls.data_bucket](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_ownership_controls) | resource |
26+
| [aws_s3_object.data_directory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_object) | resource |
27+
| [random_id.external_id](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
28+
| [random_string.suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource |
29+
30+
## Inputs
31+
32+
| Name | Description | Type | Default | Required |
33+
|------|-------------|------|---------|:--------:|
34+
| <a name="input_environment"></a> [environment](#input\_environment) | Environment name (e.g., dev, staging, prod) | `string` | `"dev"` | no |
35+
| <a name="input_tags"></a> [tags](#input\_tags) | A map of tags to assign to all resources | `map(string)` | <pre>{<br/> "ManagedBy": "Terraform"<br/>}</pre> | no |
36+
37+
## Outputs
38+
39+
| Name | Description |
40+
|------|-------------|
41+
| <a name="output_bucket_name"></a> [bucket\_name](#output\_bucket\_name) | The name of the S3 bucket created for ClickHouse Cloud |
42+
| <a name="output_configuration_instructions"></a> [configuration\_instructions](#output\_configuration\_instructions) | Instructions for providing the role information to ClickHouse Cloud |
43+
| <a name="output_role_arn"></a> [role\_arn](#output\_role\_arn) | The ARN of the IAM role to provide to ClickHouse Cloud |
44+
| <a name="output_role_external_id"></a> [role\_external\_id](#output\_role\_external\_id) | The external ID to provide to ClickHouse Cloud (keep this secure) |
45+
| <a name="output_sqs_queue_url"></a> [sqs\_queue\_url](#output\_sqs\_queue\_url) | The URL of the SQS queue receiving S3 event notifications |
46+
<!-- END_TF_DOCS -->

0 commit comments

Comments
 (0)