wfe: account cache is missing invalidation after account update

[alexzorin]

Account updates aren't immediately effective. This doesn't seem like a massive problem but here's the contexts in which it's come up occasionally:

• Account key rollover followed by other operations (https://community.letsencrypt.org/t/jws-verification-error-on-account-deactivation-preceded-by-key-rollover/175509/9)
• Writing integration tests for Certbot, have to skip Boulder when testing update_account+show_account

[aarongable]

Cache invalidation here is hard -- in particular, the account cache is in-memory per-WFE, so cache invalidation in one WFE wouldn't affect other adjacent WFEs, let alone WFEs in a different data center.

That said, we could make the account cache fall back to a normal database lookup if it gets a cache hit but the found key does not validate the signature on the payload. We expect the proportion of requests signed with the wrong key to be vanishingly small, so this shouldn't be a major rollback of the performance wins the account cache brings.

[alexzorin]

Thank you.

If the performance win is mainly around verifying JWS on non-account-related operations, would it be possible to exclude from the WFE cache the following?

1. newAccount with onlyReturnExisting
2. POST-as-GET to the account URL

I would guess that these receive relatively low traffic and it would be good to avoid producing stale data on them.

[akshithg]

I opened #8795 to document the authorization-sensitive consequences of this same stale-cache root cause. The intended patch shape keeps the WFE-local account cache, but bypasses it for account-authenticated mutation/issuance paths by using the existing read-only SA client directly, plus local cache purge after account updates/key rollover.

[akshithg]

Follow-up PR is open at #8796. It keeps the WFE-local account cache, but bypasses it for account-authenticated mutation/issuance paths using the existing read-only SA client.