`history` reports duplicate version numbers for Parameter Store key

[james-valente-simplisafe]

The output of history reports duplicate version numbers for the same Parameter Store key.

Initially, I created 3 versions of the same key.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-82

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-AA

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-82

List the history of the versions. Note how the two most recent versions are both reported as version 2.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo88 my-key-82

Event		Version		Date			User
Created		1		2024-05-16 18:30:11	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 18:37:00	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 18:37:02	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

Version 1 and the initial version 2 can be referenced directly, by number.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 1

Key		Value		Version		LastModified		User
my-key-82	my-value-82	1		2024-05-16 18:30:11	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 2

Key		Value		Version		LastModified		User
my-key-82	my-value-AA	2		2024-05-16 18:37:00	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

Unfortunately, the third version cannot be referenced by number. However, it can be referenced by relative number (-1 is latest).

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 3

Error: Failed to read: secret not found

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version -1

Key		Value		Version		LastModified		User
my-key-82	my-value-82	2		2024-05-16 18:37:02	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

Now, create a fourth version of this key.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo88 my-key-82 -- my-value-BB

The history still reports the two items as having version 2, and the most recent (fourth) version is listed as version 3.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo88 my-key-82

Event		Version		Date			User
Created		1		2024-05-16 18:30:11	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 18:37:00	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 18:37:02	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		3		2024-05-16 18:49:58	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

However, a screenshot of the AWS console reports distinct versions 1, 2, 3, and 4. Note how the Last modified date column in the AWS console reports the same timestamps as found in the output of the chamber history command above.

Note also how the following command fetches version 3 (according to Chamber), but in fact, the timestamp corresponds to Version 4 within the AWS console. As further confirmation, the value corresponds to the fourth version of the key, as modified by the chamber write command earlier in this issue description.

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 read foo/foo88 my-key-82 --version 3

Key		Value		Version		LastModified		User
my-key-82	my-value-BB	3		2024-05-16 18:49:58	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

Therefore, it would seem that Chamber may have a bug in how it tracks/reports version numbers of a Parameter Store key's history.

Although I have not been able to reproduce this, I searched previous Issues in this repo and found issue #308, which seems to report a similar bug.

[james-valente-simplisafe]

Actually, I am able to reproduce this bug, if I create and update the key/values in quick succession. Note the Date column timestamps relative to the duplicate version 2 reported in history outputs below.

Here, the delay between updates is short (2 or 3 seconds).

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-71

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-AA

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-BB

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-71 -- my-value-CC

$  AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo77 my-key-71

Event		Version		Date			User
Created		1		2024-05-16 20:24:59	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 20:25:01	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 20:25:04	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		3		2024-05-16 20:25:06	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

Another example with short delay between updates (2 seconds). Note how 3 of the 4 versions have the same version 2.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-01

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-02

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-03

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-74 -- aaaaaaaa-04

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo77 my-key-74

Event		Version		Date			User
Created		1		2024-05-16 20:42:29	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 20:42:31	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 20:42:33	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 20:42:35	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

If the delay between updates is slightly longer (4+ seconds), Chamber reports unique version numbers for each history item.

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- alpha

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- bravo

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- charlie

$ AWS_REGION=us-east-1 CHAMBER_KMS_KEY_ALIAS=xx/foo/kms-key ./chamber-v2.14.1-linux-amd64 \
    write --verbose foo/foo77 my-key-72 -- delta

$ AWS_REGION=us-east-1 ./chamber-v2.14.1-linux-amd64 history foo/foo77 my-key-72

Event		Version		Date			User
Created		1		2024-05-16 20:26:48	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		2		2024-05-16 20:26:52	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		3		2024-05-16 20:26:58	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com
Updated		4		2024-05-16 20:27:02	arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com

[james-valente-simplisafe]

Further investigation reveals that Chamber appears to be incorrectly reporting the Description and not the Version of the each Parameter Store key's history.

The history of my-key-71 gets reported as versions 1, 2, 2, 3 but should be 1, 2, 3, 4.

$ aws --region us-east-1 ssm get-parameter-history --with-decryption --name /foo/foo77/my-key-71

{
    "Parameters": [
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891099.458,
            "Labels": [],
            "Value": "my-value-71",
            "Version": 1,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "1"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891101.833,
            "Labels": [],
            "Value": "my-value-AA",
            "Version": 2,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891104.183,
            "Labels": [],
            "Value": "my-value-BB",
            "Version": 3,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-71",
            "DataType": "text",
            "LastModifiedDate": 1715891106.422,
            "Labels": [],
            "Value": "my-value-CC",
            "Version": 4,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "3"
        }
    ]
}

The history of my-key-74 gets reported as versions 1, 2, 2, 2 but should be 1, 2, 3, 4.

$ aws --region us-east-1 ssm get-parameter-history --with-decryption --name /foo/foo77/my-key-74

{
    "Parameters": [
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892149.853,
            "Labels": [],
            "Value": "aaaaaaaa-01",
            "Version": 1,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "1"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892151.883,
            "Labels": [],
            "Value": "aaaaaaaa-02",
            "Version": 2,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892153.716,
            "Labels": [],
            "Value": "aaaaaaaa-03",
            "Version": 3,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        },
        {
            "KeyId": "alias/xx/foo/kms-key",
            "Name": "/foo/foo77/my-key-74",
            "DataType": "text",
            "LastModifiedDate": 1715892155.541,
            "Labels": [],
            "Value": "aaaaaaaa-04",
            "Version": 4,
            "LastModifiedUser": "arn:aws:sts::123456789012:assumed-role/other-admin/john.doe@example.com",
            "Policies": [],
            "Tier": "Standard",
            "Type": "SecureString",
            "Description": "2"
        }
    ]
}

[james-valente-simplisafe]

Looking at the source code, I see the chamber write command intentionally stores Chamber's notion of "version" number in the secret's Description field (link).

Additionally, I see that source code was committed on June 12, 2017, which predates the AWS Blog post (link) on October 26, 2017, announcing that Parameter Store now provides native support for unique version numbers in secrets history. Thus, Chamber's use of theDescription field to store Chamber's notion of "version" understandably predates the native support for versioning within AWS Parameter Store.

As for why Chamber is unable to increment the version number in the Description field correctly, I imagine there may be some kind of eventual consistency delay within AWS when a Parameter Store secret is overwritten. So, it may be possible that these lines in source code (link) fetch the initial secret (with Description = 1, such that multiple subsequent overwrites (within a short period of time) are resulting in Description = 2.

That said, the source code (link) in the readVersion function can probably be updated to make use of the AWS native versions, assuming a history.Version field can be accessed instead of the history.Description field. According to the AWS API documentation (link), it should be possible.