Anonymous credentials draft endpoints

[LDiazN]

This PR adds support for the anonymous credentials protocol to the Ooni API:

1. Adds ooniauth-py as a dependency
2. Adds new endpoints: /sign_credential, /manifest, /submit

Note that the new measurement /submit endpoint won't replace the old one, and for now it's meant to be used mostly during development

Also updates the fastpath so that it's able to work with the new fields

Migrations

This PR will require some database migrations

Postgres

Adds the following tables to postgres:

• ooniprobe_manifest: Describes the manifest that is reported to users when they register
• ooniprobe_server_state: Defines the key pair (secret_key, public_parameters) that is used for authentication

You can find the new models here:

backend/ooniapi/services/ooniprobe/src/ooniprobe/models.py

Line 58 in 3b3bfb2

class OONIProbeServerState(Base):

And the migrations:
https://github.com/ooni/backend/blob/userauth-dep/ooniapi/common/src/common/alembic/versions/7e28b5d17a7f_add_server_state_table_for_anonymous_.py

Clickhouse

In clickhouse we need to add the fields necessary for:

• Checking if a measurement is verified
• Running the verification offline in the fastpath machine at some point in the future when we try to implement offline verification

As an example of the changes, you can look at the clickhouse_init.sql script in fastpath:

backend/fastpath/clickhouse_init.sql

Lines 42 to 46 in 3b3bfb2

	`is_verified` Int8,
	`nym` Nullable(String),
	`zkp_request` Nullable(String),
	`age_range` Nullable(String),
	`msm_range` Nullable(String),

These are the alter table statements required to run the migration in production:

ALTER TABLE fastpath
ADD COLUMN is_verified Int8,
ADD COLUMN nym Nullable(String),
ADD COLUMN zkp_request Nullable(String),
ADD COLUMN age_range Nullable(String),
ADD COLUMN msm_range Nullable(String);

Feedback

This is still early work to define the API for the anonymous credentials protocol. Some things that could benefit from a bit of feedback are:

• Naming of the endpoints
  • /sign_credential used to be named /register but it clashes with the older /register one, so I chose a different name
• Parameters and usage
• Lifecycle of manifest and server state: Right now we always create a new manifest and server state if none is available to keep the API working, but we might want to be more deliberate with how we work with this

closes #1014 #1015

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.92%. Comparing base (d943f03) to head (5deb275).
⚠️ Report is 11 commits behind head on master.

❌ Your project check has failed because the head coverage (90.92%) is below the target coverage (95.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1016      +/-   ##
==========================================
- Coverage   92.81%   90.92%   -1.90%     
==========================================
  Files          17       28      +11     
  Lines        1281     2932    +1651     
  Branches       65      247     +182     
==========================================
+ Hits         1189     2666    +1477     
- Misses         78      206     +128     
- Partials       14       60      +46     

Flag	Coverage Δ
oonimeasurements	87.60% <ø> (?)
ooniprobe	?
oonirun	99.39% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hellais]

I left some comments on small improvements and some questions.

We can discuss on slack anything which is not clear.

[hellais]

Can we just leave this inlined? I think there is no need to have an extra line and variable.

[LDiazN]

Done!

[hellais]

Can we use ISO standard? Make sure it has in it T and Z to indicate timezone.

[LDiazN]

I'm not sure if that's a good idea because ts is used to compute the measurement id:

backend/ooniapi/services/ooniprobe/src/ooniprobe/routers/v1/probe_services.py

Line 762 in c40e14b

msmt_uid = f"{ts}_{cc}_{test_name}_{h}"

So we would be changing the measurement id format. I'm not too confident that this change won't break anything else. What do you think?

[hellais]

Yeah in that case we can't do it. Leave leave this as-is.

[hellais]

Is this necessary? cc @mmaker