Add ADM setting validation

[jrconlin]

We have an increasingly complex settings engine for ADM. It is absolutely possible for mistakes to be made while specifying ADM settings (e.g.

{
   "host": "example.com",
   "paths": [
        {"value": "/", "matching": "prefix" },
        {"value": "/path", "matching": "exact"}
    ]
}

We should reject settings that have these sort of conflicts:

• "empty" prefix paths
• overlapping prefix and exact paths (should we allow both prefix and exact paths?)
• missing terminal / for prefix matches
• etc.

Will also need to determine failure policy (e.g. how errors are reported out and what actions to take when encountering errors.)

┆Issue is synchronized with this Jira Task
┆epic: Simplify Contile Settings Management

[data-sync-user]

➤ Nive Suresh commented:

JR Conlin and Philip Jenvey to sync offline.

[data-sync-user]

➤ JR Conlin commented:

A few follow up notes:

1. Contile will perform some of these checks automatically (e.g. it will reject prefix paths that do not end with a /. (When reading from a Storage Bucket, we note an error, but do not accept the new settings.)

2. With some of the work around automating settings, this will probably be handled by an external tool or system. (IOW: try to catch these sorts of errors on generation rather than on-the-fly or at the server)

[data-sync-user]

➤ JR Conlin commented:

This might be a good project as a cloud function ( https://cloud.google.com/functions ) (potentially triggered off of a cloud storage write ( https://cloud.google.com/functions/docs/calling/storage ) to:

1. Read data in JSON format from a Cloud Storage Bucket (data will be the ADM component of settings, see https://github.com/mozilla-services/contile/blob/main/integration-tests/volumes/contile/adm_settings.json ( https://github.com/mozilla-services/contile/blob/main/integration-tests/volumes/contile/adm_settings.json ) for an example)

2. the DEFAULT advertiser is special, in that it is used to specify the default values for other strings. Alterations to this field should be allowed, but warnings would be very useful. (We may wish to keep a separate bucket that contains this as a template value, along with any other settings)

3. Advertiser URLs are landing URLs for users once they click on a given link. We need to be very explicit about what we allow here to prevent users from being directed to something unexpected. The following are general rules we have in place, but feel free to add additional rules if desired (e.g. urldecode paths to flag things like foo%2Fbar) If you have any questions, feel free to ask.
   verify advertiser urls are a list of dictionaries containing “host” and an optional “paths”
   1. hosts should be valid RFC1123 ( https://datatracker.ietf.org/doc/html/rfc1123 ) (although we don’t have to check if the host actually exists, but it might be nice.)
   2. hosts should not include protocols (e.g. https), port numbers (e.g. :80) or paths (e.g. /abc123)
   3. the optional paths should contain a list of dictionaries

   1. each dictionary should contain a value field indicating the path that should be used.
   2. the value should begin with a /
   3. each dictionary should include a matching type of either prefix or exact.
   4. a prefix matching type should end with a /
   5. path value field for a given advertiser_url set should not overlap (e.g. /foo and /foo/bar would be an error)
   6. path prefix matching type should not have a value of /

4. include_regions should only list allowed regions.

5. Empty lists can be optionally dropped.

The list of errors generated should include the location of the error, a statement of the error, and any suggested fixes. Presume the recipient of the final report is not technical, so ideally this should be as friendly as possible. Bonus points for including error numbers so that additional information about this error could be cross referenced to documentation later.