INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.
Add test for "Encrypted fields found" error (ensure this exception still happens)
Add check for model schema not matching encrypted fields
Document key_vault_namespace must be encrypted db
Document that fields within EmbeddedModelArrayField can't be encrypted
Document workflow:
- Develop and test local
- Deploy and run migrate
- Run showencryptedfieldsmap and set encrypted_fields_map in auto_encryption_opts
- Deploy again

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

Via https://github.com/mongodb-labs/drivers-evergreen-tools/tree/master/.evergreen/csfle

…ove mongocryptd

… it part of the normal flow

Move crypt shared installation instructions to "Installation" section and crypt shared configuration steps to "Configuring the DATABASES setting" section.

Fall back to local if no env vars set

timgraham · 2025-12-09T02:59:25Z

.github/workflows/encrypted_settings.py

+    KMS_CREDENTIALS = {"aws": {"key": _AWS_KEY_ARN, "region": _AWS_REGION}}
+else:
+    KMS_PROVIDERS = {"local": {"key": os.urandom(96)}}
+    KMS_CREDENTIALS = {"local": {}}


As far as I know "local" isn't required in this dictionary and I wouldn't put it there so we don't test an unrealistic scenario.

timgraham · 2025-12-09T03:00:24Z

.github/workflows/encrypted_settings.py

+    _AWS_REGION = os.environ.get("FLE_AWS_KMS_REGION", "us-east-1")
+    _AWS_KEY_ARN = os.environ.get(
+        "FLE_AWS_KMS_KEY_ARN",
+        "arn:aws:kms:us-east-1:579766882180:key/89fcc2c4-08b0-4bd9-9f25-e30687b580d0",
+    )


Should there be fallback values? Seems it might be better to fail loudly than silently.

timgraham · 2025-12-09T03:02:45Z

.github/workflows/encrypted_settings.py

+    KMS_PROVIDERS = {"aws": AWS_CREDS}
+    KMS_CREDENTIALS = {"aws": {"key": _AWS_KEY_ARN, "region": _AWS_REGION}}


I would caution against using uppercased values that look like Django settings. (Any values in this file are accessible as such. It could lead to inadvertent referencing of them.)

timgraham · 2025-12-09T03:04:18Z

tests/encryption_/test_management.py

+            master_key = connections["encrypted"].settings_dict["KMS_CREDENTIALS"][
+                self.DEFAULT_KMS_PROVIDER
+            ]


Since we only support on provider, I think you could avoid all the logic in test_base.py and simply retrieve the configured provider/master key values similar to how schema.py does it: next(iter(kms_providers.keys()))

timgraham · 2025-12-09T03:06:23Z

.evergreen/run-tests.sh

+# Export secrets as environment variables
+. ../secrets-export.sh


This file is from driver tools or something?

timgraham · 2025-12-09T03:07:50Z

.evergreen/run-tests.sh

-pip install -e .
+pip install -e '.[encryption]'


Try to avoid encryption related tasks like this on non-encrypted builds.

timgraham · 2025-12-09T03:09:25Z

.evergreen/config.yml

+    display_name: Run Tests 8.2 QE
+    run_on: rhel87-small
+    expansions:
+      MONGODB_VERSION: "8.2"


My impression is that we only want to test with dot zero MongoDB's, but lets get clarification on this.

timgraham · 2025-12-09T03:11:40Z

.github/workflows/encrypted_settings.py

+    "accessKeyId": os.environ.get("FLE_AWS_KEY", ""),
+    "secretAccessKey": os.environ.get("FLE_AWS_SECRET", ""),


Comment where the environment variable names come from given that then name doesn't appear anywhere else in this repo. (Is it really configured correctly?) Maybe it's better to have some configuration variable like "USE_AWS_KMS" rather than inferring it from these values.

I thought it might be incorrect since the names are different from this line in evergreen config: include_expansions_in_env: ["AWS_SECRET_ACCESS_KEY", "AWS_ACCESS_KEY_ID", "AWS_SESSION_TOKEN"]

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life force-pushed the INTPYTHON-527 branch 2 times, most recently from c89be3d to b22dfbd Compare December 7, 2025 21:48

WaVEV and others added 24 commits December 7, 2025 16:49

Remove $facet in top level group stages

9fbd5c9

Wrap query for global aggregation

5cc34c7

[temp] remove Django tests for faster CI

4072978

INTPYTHON-527 Add Queryable Encryption support

fed63b2

[wip] updates to field docs

32ee2be

add QuerySet tests

724af49

use shared library instead of mongocryptd

1535415

adjust message and add test for missing auto_encryption_opts error

07d51bb

polish howto

ccd33cb

doc query limitations + docs polish + todos

200bdcb

edit "Dynamic library path configuration"

43ab91f

combine topic guide with howto

3384470

Remove QE from 5.2.0 beta 2 release notes

60e4ba3

Add "start csfle servers" func to evergreen config

377d683

Via https://github.com/mongodb-labs/drivers-evergreen-tools/tree/master/.evergreen/csfle

remove support for multiple kms providers

180e9d6

fix join tests following foreignField/localField refactor

234a9aa

Add tests-8-qe to evergreen buildvariants

05753a1

fix less than lookup on encrypted fields

5edac6b

simplify "Configuring the Automatic Encryption Shared Library" to rem…

8b32497

…ove mongocryptd

reorder "Configuring the Automatic Encryption Shared Library" to make…

6897d55

… it part of the normal flow

update docs/tests for $facet removal

e8f0c44

Combine crypt shared w/installation & db setup

592dd7f

Move crypt shared installation instructions to "Installation" section and crypt shared configuration steps to "Configuring the DATABASES setting" section.

versionadded updated from 5.2.3 to 6.0.0.

c760b5a

Configure AWS KMS for testing on evergreen

76200b1

Fall back to local if no env vars set

aclark4life force-pushed the INTPYTHON-527 branch from b22dfbd to 76200b1 Compare December 7, 2025 21:49

aclark4life added 2 commits December 7, 2025 17:13

Move start csfle servers to pre

2f4c4be

doc edits

1075c9f

timgraham reviewed Dec 9, 2025

View reviewed changes

		KMS_PROVIDERS = {"aws": AWS_CREDS}
		KMS_CREDENTIALS = {"aws": {"key": _AWS_KEY_ARN, "region": _AWS_REGION}}

		# Export secrets as environment variables
		. ../secrets-export.sh

		"accessKeyId": os.environ.get("FLE_AWS_KEY", ""),
		"secretAccessKey": os.environ.get("FLE_AWS_SECRET", ""),

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited by timgraham Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timgraham Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aclark4life commented Jun 27, 2025 •

edited by timgraham

Loading

timgraham Dec 9, 2025 •

edited

Loading