feat!: rebuild proxy with multistore and cloudflare workers runtime#116
Merged
feat!: rebuild proxy with multistore and cloudflare workers runtime#116
Conversation
currently reducing the data proxy to read-only anonymous access
alukach
force-pushed
the
refactor/multistore-readonly
branch
from
48bb10d to
0d9a80e
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
7 tasks
…ath-mapping crates Replace 6 boilerplate source files (worker_backend, worker_infra, fetch_connector, noop_creds, tracing_layer, routing) and their test with imports from the new multistore-cf-workers and multistore-path-mapping crates. The XML list rewriting is now handled by multistore core via MappedRegistry's display_name and ListRewrite. Files deleted: - src/worker_backend.rs (WorkerBackend -> multistore-cf-workers) - src/worker_infra.rs (JsBody, response helpers -> multistore-cf-workers) - src/fetch_connector.rs (FetchConnector -> multistore-cf-workers) - src/noop_creds.rs (NoopCredentialRegistry -> multistore-cf-workers) - src/tracing_layer.rs (WorkerSubscriber -> multistore-cf-workers) - src/routing.rs (path parsing -> multistore-path-mapping + simplified lib.rs) - tests/routing.rs (covered by multistore-path-mapping tests) Source reduced from 9 files / ~1,520 lines to 3 files / ~700 lines. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract request classification logic from lib.rs into src/routing.rs - Add comprehensive tests in tests/routing.rs (31 tests total) - Move wasm-only deps to target-specific dependencies so tests run natively - Remove unused _method param from classify_request - Add request-level debug logging Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
alukach
changed the title
|
🚀 Latest commit deployed to https://source-data-proxy-pr-116.source-coop.workers.dev
|
alukach
changed the title
This was referenced Mar 24, 2026
alukach
pushed a commit
that referenced
this pull request
Mar 26, 2026
🤖 I have created a release *beep* *boop* --- ## [2.0.0](v1.1.0...v2.0.0) (2026-03-26) ### ⚠ BREAKING CHANGES * rebuild proxy with multistore and cloudflare workers runtime ([#116](#116)) ### Features * add Analytics Engine request logging ([#119](#119)) ([d9ab62f](d9ab62f)) * rebuild proxy with multistore and cloudflare workers runtime ([#116](#116)) ([3e07478](3e07478)) ### Bug Fixes * deserialize before caching ([295b965](295b965)) * Return 502 on API error instead of empty list ([d0ff1f0](d0ff1f0)) * scrutinise list type more closely ([d63e4f8](d63e4f8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: source-release-bot[bot] <265100246+source-release-bot[bot]@users.noreply.github.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I'm changing
Pushed by a recent spike of high ALB egress bills, this PR swaps out the data proxy for a data proxy written with multistore. This allows us to deploy the data proxy onto Cloudflare Workers, thereby pushing all egress charges directly to S3 and in line with the AWS Open Data Program.
This is a read-only proxy, write operations will be added at a later date.
How I did it
Deployed to Cloudflare Workers. We're currently serving ~4M requests per day and are seeing an error rate of ~0.001%
Custom URLs
Obtaining custom URLs without migrating all of the source.coop DNS settings over the Cloudflare was a bit of a challenge. I opted to host the proxy workers on
coolnewgeo.com(an unused domain owned by Radiant Earth):data.coolnewgeo.com- prodstaging.data.coolnewgeo.com- stagingCustom Hostnames have been set up under
coolnewgeo.comfordata.source.coopanddata.staging.source.coop, both pointing to a null fallback origin offallback.coolnewgeo.com(which has a DNS A record pointing to192.0.2.1).Configured routes on the workers connect these custom hostnames to each worker environment:
data.source.coop/wrangler.toml
Lines 5 to 11 in 0a44d6b
data.source.coop/wrangler.toml
Lines 36 to 40 in 0a44d6b
How to test it
This has been running in production for the past week and is used by https://source.coop.
PR Checklist
This PR has no breaking changes.and I have opened issue/PR #XXX to track the change.
Related Issues
TODO
main