Added pod affinity if the default ReadWriteOnce is used.

[keskival]

If accessMode ReadWriteOnce is used, then the volume can only be accessed in a write mode from a single node. This means that all pods which require write access to the volume need to be co-located to the same node. Failing that typically leads to a difficult to untangle locked volume mount situation where containers are stuck in ContainerCreating.

This change adds the necessary pod affinities if the related access modes are set in the values.yaml.

[keskival]

I am in progress of testing this, putting it here already for transparency.
Note that this relates to the issue here: #17

[SISheogorath]

We literally just removed this in order to reduce unexpected affinities by just selecting a different volume access mode.

I guess the question becomes towards what end of the spectrum we want tonstart optimising the chart. Do we want to keep it generic and unopinionated or move towards an ideal, predefined setup?

See this conversation: mastodon/mastodon#20733 (comment)

[keskival]

Ok fair enough, but the default configuration isn't functional, and the jobs have analogous affinities set up.

In any case, this has been tested as far as template generation goes and produces templates in the expected form.

If ReadWriteOnce is used, not having a pod affinity would be an error which causes a catastrophic, random lock up.

I am fine with adding a visible warning to the values.yaml about this issue as well.

This is now confirmed working in my own cluster as well. Previously the defaults caused it to become locked up in a difficult to understand manner, which is what this PR fixes.

Added S3 conditioned disabling as well.