`aws cloudformation package` breaks certain string literals in templates

[idm-ryou]

CloudFormation template

template.yml

Parameters:
  Value7:
    Type: String
    Default: "077777777777"
  Value8:
    Type: String
    Default: "088888888888"
  Value9:
    Type: String
    Default: "099999999999"
Resources:
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ./template_child.yml

template_child.yml

# This template is not so important
Resources:
  Dummy:
    Type: Custom::Whatever
    Properties:
      ServiceToken: whatever
      Value1: whatever

Commands to reproduce

$ uname -a
Darwin xxxxxxx 18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64

$ aws --version
aws-cli/1.16.110 Python/3.7.2 Darwin/18.2.0 botocore/1.12.100

$ aws cloudformation package --s3-bucket BUCKET_NAME --template-file template.yml                                                           
Uploading to 137ce8b72427772da39a43ddc087908a.template  101 / 101.0  (100.00%)
Parameters:
  Value7:
    Type: String
    Default: '077777777777'
  Value8:
    Type: String
    Default: 088888888888
  Value9:
    Type: String
    Default: 099999999999
Resources:
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/BUCKET_NAME/137ce8b72427772da39a43ddc087908a.template

Quotes of Value8 and Value9 are removed after aws cloudformation package.

It seems that leads to the values interpreted as numbers during stack operation.
If I use the values for !Sub, values will be like 9.9999999999E10, which is unintended and cause errors.

[idm-ryou]

https://github.com/aws/aws-cli/blob/develop/awscli/customizations/cloudformation/yamlhelper.py

$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> from awscli.customizations.cloudformation.yamlhelper import yaml_parse, yaml_dump

>>> parsed = yaml_parse('sevens: "077777777777"\neights: "088888888888"')

>>> parsed
OrderedDict([('sevens', '077777777777'), ('eights', '088888888888')])

>>> yaml_dump(parsed)
"sevens: '077777777777'\neights: 088888888888\n"

It seems quotes are erased during dumping to yaml.

$ python3
Python 3.7.2 (default, Feb 12 2019, 08:15:36)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml

>>> yaml
<module 'yaml' from '/usr/local/Cellar/awscli/1.16.110/libexec/lib/python3.7/site-packages/yaml/__init__.py'>

>>> yaml.dump({"sevens": "077777777777", "eights": "088888888888"}, default_flow_style=False)

"eight: 088888888888\nseven: '077777777777'\n"

Seems a bug of PyYAML?:thinking:

[idm-ryou]

I thought PyYAML generates invalid yaml, but PyYAML parses correctly (as string type) without quotes.

>>> yaml.load("eights: 088888888888\nsevens: '077777777777'\n")
{'eights': '088888888888', 'sevens': '077777777777'}

I suspect either PyYAML or the yaml parser used in CloudFormation API is not following the yaml spec (related to octal base int type and string type without quotes) and causing this issue.

https://yaml.org/type/int.html

In any case, aws cloudformation package should generates yaml that can be used for CloudFormation.

[idm-ryou]

Any updates?
Please let me know if you need further information.

[idm-ryou]

I guessed that CloudFormation API uses SnakeYAML from its behavior, so I filed an issue for SnakeYAML.

https://bitbucket.org/snakeyaml/snakeyaml/issues/442

The SnakeYAML author says that it is a bug in YAML 1.1 spec...

I think aws-cli should dump strings in YAML with quotes for parser interoperability.
https://stackoverflow.com/questions/38369833/pyyaml-and-using-quotes-for-strings-only

[idm-ryou]

Related?
yaml/pyyaml#98
awslabs/aws-cfn-template-flip#41
aws/aws-sam-cli#867

[justnance]

We are looking into this issue. @sanathkr, what are your thoughts?

[beibeiyang]

We are running into a similar issue. We have Account ID as the key in template.yaml:

template.yaml

...
Mappings:
  AccountMap:
    "098111111198":
      Env: "dev"
    "988111111198":
      Env: "test"
    "123456789012":
      Env: "prod"
...

After the sam package --template-file template.yaml --output-template-file packaged.yaml --s3-bucket BUCKET_NAME, quotes around the first Account ID got stripped:

packaged.yaml

...
Mappings:
  AccountMap:
    098111111198:
      Env: dev
    '988111111198':
      Env: test
    '123456789012':
      Env: prod
...

The subsequent aws cloudformation deploy --template-file packaged.yaml --stack-name STACK_NAME ... thinks 098111111198 is a number and throws An error occurred (ValidationError) when calling the CreateChangeSet operation: Template format error: [/Mappings/AccountMap] map keys must be strings; received numeric [9.8111111198E10] instead.

$ sam --version
SAM CLI, version 0.17.0
$ aws --version
aws-cli/1.16.150 Python/3.7.3 Darwin/18.5.0 botocore/1.12.140
$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.9.172'

We would appreciate suggestions to any workarounds. Thanks!

[jiravani]

Having the exact same issue outlined by @beibeiyang above. Any updates?