Input and integration packages allow any file at the root of the package

[mrodm]

Currently, input and integration packages allow to define any file or folder at the root of the working copy.

• integration:
  

  package-spec/spec/integration/spec.yml

  Line 7 in 18af7f0

  additionalContents: true

• input:
  

  package-spec/spec/input/spec.yml

  Line 7 in 18af7f0

  additionalContents: true

Content packages are the only ones that disallow that.

package-spec/spec/content/spec.yml

Line 7 in 18af7f0

spec:

Testing to disallow those extra files cause that current packages fail the validation with files or folders not allowed. Some examples found (not a full list):

1. item [img] is not allowed in folder [/home/user/Coding/work/integrations/packages/zscaler_zpa]
1. item [img] is not allowed in folder [/home/user/Coding/work/integrations/packages/ded]
1. item [img] is not allowed in folder [/home/user/Coding/work/integrations/packages/modsecurity]
2. item [kibana] is not allowed in folder [/home/user/Coding/work/integrations/packages/http_endpoint]
1. item [lifecycle.yml] is not allowed in folder [/home/user/Coding/work/integrations/packages/extrahop]
1. item [fields] is not allowed in folder [/home/user/Coding/work/integrations/packages/dga]
2. item [img] is not allowed in folder [/home/user/Coding/work/integrations/packages/dga]
1. item [kibana] is not allowed in folder [/home/user/Coding/work/integrations/packages/tcp]
2. item [system-test-note.md] is not allowed in folder [/home/user/Coding/work/integrations/packages/redisenterprise]
1. item [DEVELOPMENT.md] is not allowed in folder [/home/user/Coding/work/integrations/packages/tychon]
1. item [fields] is not allowed in folder [/home/user/Coding/work/integrations/packages/problemchild]
2. item [system-test-note.md] is not allowed in folder [/home/user/Coding/work/integrations/packages/iis]
...

A summary of the errors found:

• img folder defined in integration packages using spec version lower than 3.2.2
  • Related to Initial definition of content pack[age]s #777 and Prepare release 3.2.2 #793
  • Most of the errors are related to this folder being present in integrations packages with spec < 3.2.2
• kibana folder defined in input packages
  • input packages should not contain any kibana folder
  • how does Fleet behave if this folder is defined?
• fields folder defined in integration packages.
  • in integration packages, fields should be under data stream folders.
• extra markdown files: DEVELOPMENT.md, system-test-note-md
  • should they be under docs/? should they be allowed ?
• extra yaml files: lifecycle.yml
  • maybe just a leftover from some PR ?

Tested via #992
Packages from integrations repository tested in elastic/integrations#15500 (more than 250 packages failing in this test).

One option is to apply this change starting at a specific spec version.
In any case, it is needed to assess how to proceed with all these packages with extra files or folders if this change is applied.

[jsoriano]

Thanks for the summary of errors found!

• Most of the errors are related to this folder being present in integrations packages with spec < 3.2.2

We could set additionalContents: false only for packages with format version over 3.3.0, or better backport the definition for img, as it should be definitely supported in older versions.

kibana folder defined in input packages

• input packages should not contain any kibana folder
• how does Fleet behave if this folder is defined?

Fleet probably installs these assets, I don't think there is any code branch depending on package type there. We could try though.

We could check the input packages with kibana folder in case it makes sense to support some assets.

• fields folder defined in integration packages.
  • in integration packages, fields should be under data stream folders.

This looks wrong, though we should check if Fleet has some behavior implemented for that.

extra markdown files: DEVELOPMENT.md, system-test-note-md

• should they be under docs/? should they be allowed ?

Maybe this is the reason to allow additional contents in the root. Packages could contain additional documentation, license or notice files. But maybe if the only cases we have are about documentation, we could rather put them under docs, or somewhere under _dev if they are only for development.

extra yaml files: lifecycle.yml

• maybe just a leftover from some PR ?

This is supported in input packages and in data streams. I see the extrahop package has it, and was added when adding a data stream, it should be probably moved to this data stream.

One option is to apply this change starting at a specific spec version.

Yes, I would do that to avoid needing to make changes in very old packages, but maybe we could still apply the change for 3.0.

For every case we should the behavior in Fleet, if it makes sense, adjust the spec to allow it. After this, re-evaluate.

For cases that should not be allowed, fix the package. If we find many packages on this situation we could reconsider the spec version to make the change, to avoid having problems with old packages, backports and so on.

If this affects too many packages, maybe we could consider moving this to 4.0 as a breaking change in the spec. But it looks like the kind of errors could be addressed by small changes in the spec or fixes in packages.