diff --git a/elastio-s3-changelog/README.md b/elastio-s3-changelog/README.md index 311731f..6178ed7 100644 --- a/elastio-s3-changelog/README.md +++ b/elastio-s3-changelog/README.md @@ -1,7 +1,7 @@ # S3 Changelog For large buckets, scanning all objects might take a lot of time. This CloudFormation template enables -the Changelog feature for S3 buckets, which significantly improves the scan performance after the initial +the Changelog feature for S3 buckets, which improves the scan performance after the initial scan of the entire bucket is done. This template deploys an SQS queue and an EventBridge rule which sends S3 update events to the queue. @@ -32,7 +32,16 @@ Then, the Elastio `iscan` job reads those events to perform the scanning of new 3. Fill in the main parameters: * *BucketNames* - comma-separated list of S3 bucket names; - * *ScanExistingObjects* - set to `true` if you want to perform the initial scan of all objects in the bucket(s); + * *CrossAccount* - set this to true if the scan happens in a different AWS account than the + S3 bucket (i.e. you are using a centralized scanning account). If it is not set + to true in the cross-account case, an error about missing AWS Lambda functions + will occur during stack creation. + + * *ScanExistingObjects* - if set to true, the full initial scan of the bucket will be performed. + If set to false, only new objects in the bucket will be scanned. + This option can only be enabled when CrossAccount is false. If it is not set + to false in the cross-account case, an error about missing AWS Lambda functions + will occur during stack creation. * *KeyPrefixes* - (optional) comma-separated list of prefixes of objects to scan. This will be applied to all buckets. If you want to use different prefixes for different buckets, you need to deploy multiple S3 Changelog stacks. diff --git a/elastio-s3-changelog/cloudformation-multiple-buckets-with-macros.yaml b/elastio-s3-changelog/cloudformation-multiple-buckets-with-macros.yaml index 0e0a62a..cd70972 100644 --- a/elastio-s3-changelog/cloudformation-multiple-buckets-with-macros.yaml +++ b/elastio-s3-changelog/cloudformation-multiple-buckets-with-macros.yaml @@ -10,6 +10,9 @@ Transform: Parameters: BucketNames: Type: String + CrossAccount: + Type: String + Default: 'false' ScanExistingObjects: Type: String Default: 'false' @@ -39,6 +42,7 @@ Resources: Properties: Parameters: BucketName: !Ref BucketName + CrossAccount: !Ref CrossAccount ScanExistingObjects: !Ref ScanExistingObjects KeyPrefixes: !Ref KeyPrefixes DisableEventCollection: !Ref DisableEventCollection diff --git a/elastio-s3-changelog/cloudformation-multiple-buckets.yaml b/elastio-s3-changelog/cloudformation-multiple-buckets.yaml index 62bd438..c56cf89 100644 --- a/elastio-s3-changelog/cloudformation-multiple-buckets.yaml +++ b/elastio-s3-changelog/cloudformation-multiple-buckets.yaml @@ -12,6 +12,16 @@ Parameters: Comma-delimited list of S3 bucket names. Example: examplebucket1, examplebucket2 + CrossAccount: + Type: String + AllowedValues: ['true', 'false'] + Default: 'false' + Description: > + Set this to true if the scan happens in a different AWS account than the + S3 bucket (i.e. you are using a centralized scanning account). If it is not set + to true in the cross-account case, an error about missing AWS Lambda functions + will occur during stack creation. + ScanExistingObjects: Type: String AllowedValues: ['true', 'false'] @@ -19,6 +29,9 @@ Parameters: Description: > If set to true, the full initial scan of the bucket will be performed. If set to false, only new objects in the bucket will be scanned. + This option can only be enabled when CrossAccount is false. If it is not set + to false in the cross-account case, an error about missing AWS Lambda functions + will occur during stack creation. KeyPrefixes: Type: CommaDelimitedList @@ -75,6 +88,7 @@ Metadata: - Label: { default: Main Parameters } Parameters: - BucketNames + - CrossAccount - ScanExistingObjects - KeyPrefixes @@ -95,6 +109,7 @@ Resources: Properties: Parameters: BucketNames: !Join [',', !Ref BucketNames] + CrossAccount: !Ref CrossAccount ScanExistingObjects: !Ref ScanExistingObjects KeyPrefixes: !Join [',', !Ref KeyPrefixes] DisableEventCollection: !Ref DisableEventCollection diff --git a/elastio-s3-changelog/cloudformation-single-bucket.yaml b/elastio-s3-changelog/cloudformation-single-bucket.yaml index 4db4875..1a61da6 100644 --- a/elastio-s3-changelog/cloudformation-single-bucket.yaml +++ b/elastio-s3-changelog/cloudformation-single-bucket.yaml @@ -9,6 +9,16 @@ Parameters: ConstraintDescription: Bucket name must be at least 3 characters long. Description: S3 bucket name. + CrossAccount: + Type: String + AllowedValues: ['true', 'false'] + Default: 'false' + Description: > + Set this to true if the scan happens in a different AWS account than the + S3 bucket (i.e. you are using a centralized scanning account). If it is not set + to true in the cross-account case, an error about missing AWS Lambda functions + will occur during stack creation. + ScanExistingObjects: Type: String AllowedValues: ['true', 'false'] @@ -16,6 +26,9 @@ Parameters: Description: > If set to true, the full initial scan of the bucket will be performed. If set to false, only new objects in the bucket will be scanned. + This option can only be enabled when CrossAccount is false. If it is not set + to false in the cross-account case, an error about missing AWS Lambda functions + will occur during stack creation. KeyPrefixes: # Unfortunately, it's impossible to pass CommaDelimitedList to a nested stack, @@ -74,6 +87,7 @@ Metadata: - Label: { default: Main Parameters } Parameters: - BucketName + - CrossAccount - ScanExistingObjects - KeyPrefixes @@ -89,6 +103,7 @@ Metadata: - QrtsMaxDelay Conditions: + NotCrossAccount: !Equals [!Ref CrossAccount, 'false'] ScanExistingObjects: !Equals [!Ref ScanExistingObjects, 'true'] DisableEventCollection: !Equals [!Ref DisableEventCollection, 'true'] EnableDlq: !Equals [!Ref EnableDlq, 'true'] @@ -216,6 +231,7 @@ Resources: queueRefreshSchedule: Type: AWS::Scheduler::Schedule + Condition: NotCrossAccount Properties: Description: A schedule to invoke the periodic changelog queue refresh lambda ScheduleExpression: cron(0/15 * * * ? *) diff --git a/elastio-s3-changelog/version b/elastio-s3-changelog/version index 626799f..8c1384d 100644 --- a/elastio-s3-changelog/version +++ b/elastio-s3-changelog/version @@ -1 +1 @@ -v1 +v2