feat(sdk): TinyGo WASM canary builds and spike plan#3063
feat(sdk): TinyGo WASM canary builds and spike plan#3063pflynn-virtru wants to merge 24 commits intomainfrom
Conversation
… and zipwrite modules
ADR documenting the TinyGo hybrid WASM architecture spike (SDK-WASM-1), including host crypto ABI, go/no-go criteria, and task breakdown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Summary of ChangesHello @pflynn-virtru, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request lays the groundwork for a significant architectural shift by exploring the feasibility of a TinyGo-compiled WebAssembly (WASM) core engine for TDF operations. It introduces a set of targeted canary programs to assess TinyGo's compatibility with crucial Go standard library components and establishes a detailed spike plan. The primary aim is to validate a hybrid approach where cryptographic primitives are delegated to the host environment, while core TDF logic resides within a compact and portable WASM module, paving the way for enhanced flexibility and deployment options. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Go code, now WASM bound, TinyGo makes it small and fast, New frontiers are found. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive spike plan (ADR) for a TinyGo-based WASM core engine and adds several canary programs to test the compatibility of necessary Go standard library packages with TinyGo. The ADR is well-structured and detailed, covering architecture, ABI, risks, and a task breakdown. The canary programs are good initial steps to de-risk the effort.
My review focuses on improving the clarity of the ADR and addressing a potential memory safety issue in the wasmimport canary. I've suggested a clarification in the ADR's objective to align it better with the documented scope. I've also recommended adding a crucial comment to the wasmMalloc implementation to highlight its dependency on the -gc=leaking build flag for memory safety, which is critical for future maintenance.
There was a problem hiding this comment.
The current implementation of wasmMalloc has a potential memory safety issue. The buf slice is allocated within the function, and returning a pointer to its underlying data can lead to a dangling pointer if the garbage collector reclaims buf after the function returns. While this is mitigated by compiling with the -gc=leaking flag as mentioned in the ADR, this critical dependency is not apparent from the code itself. For better maintainability and to prevent accidental misuse, I recommend adding a comment explaining why this is safe under the specific build conditions and highlighting the intentional memory leak.
buf := make([]byte, size)
// NOTE: This leaks memory. This is only safe because the module is compiled
// with the `-gc=leaking` flag, which prevents the garbage collector from
// reclaiming the memory.
return uint32(uintptr(unsafe.Pointer(&buf[0])))| ## Objective | ||
|
|
||
| Validate that a TinyGo-compiled WASM module can perform TDF3 single-segment | ||
| encrypt/decrypt with all crypto delegated to host functions, producing output |
There was a problem hiding this comment.
The objective states encrypt/decrypt, but the scope of the spike seems to be focused on implementing encryption within the WASM module and then validating the output using the existing Go SDK for decryption. The 'Explicitly Out of Scope' section also mentions 'Decrypt inside WASM' is for a future milestone (M2). To better reflect the spike's goal, consider clarifying that only encryption will be performed by the WASM module by removing /decrypt.
| encrypt/decrypt with all crypto delegated to host functions, producing output | |
| encrypt with all crypto delegated to host functions, producing output |
X-Test Failure Report |
…e logic - Removed `wasmimport` module and redundant crypto directives. - Consolidated `calculateSignature` logic into `writer.go` and removed it from `manifest.go`. - Updated TinyGo canary workflow to reflect WASM module restructuring.
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Adds self-contained decrypt benchmarks that construct valid TDFs programmatically and inject payload keys directly, bypassing KAS. Covers 1MB–2GB WriteTo path and 100MB streaming Read() path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Benchmarks for the experimental streaming TDF Writer covering: - End-to-end encrypt (NewWriter + WriteSegment + Finalize) - Single segment encrypt throughput (WriteSegment only) - Full TDF assembly (segments + finalize bytes) Sizes: 1MB, 100MB, 1GB (short-mode skippable). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn@virtru.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…ibility
Replace reflection-based encoding/json with tinyjson (CosmWasm fork) codegen
for manifest, assertion, and policy structs in the TDF write path. This is a
prerequisite for the WASM core engine spike (SDK-WASM-1) since encoding/json
panics at runtime under TinyGo.
Changes:
- Add tinyjson codegen for manifest.go (13 types) and assertion_types.go (3 types)
- Change KeyAccess.PolicyBinding from interface{} to concrete PolicyBinding type
- Replace json.Marshal with .MarshalJSON() in writer.go, key_access.go, assertion.go
- Move Assertion/Statement/Binding structs to assertion_types.go for codegen
- Drop polymorphic Statement.Value UnmarshalJSON (reader concern, out of WASM scope)
- Add tinyjson TinyGo canary with manifest/policy/assertion round-trip validation
- Add wasm/Makefile with toolcheck, build, run, generate targets
- Add wasm/README.md with TinyGo/tinyjson/wasmtime install instructions
Canary results: tinyjson module compiles to 62KB raw / 29KB gzipped WASM,
all round-trip tests pass under wasmtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…r WASM Copy production zipstream writer code (5 files from sdk/internal/zipstream/) into a standalone canary module and verify it compiles and runs correctly under TinyGo WASM. This completes Phase 1 (Foundation) of the WASM core engine spike (SDK-WASM-1). The canary exercises: - Single-segment TDF ZIP creation (header + manifest + central directory) - Multi-segment out-of-order writing (3 segments in order 2, 0, 1) - ZIP64 mode (ZIP64 EOCD + locator signatures) - CRC32 combine (multi-part checksum matches direct computation) Key finding: time.Time and time.Now() work correctly under TinyGo — the only identified risk for zipstream compatibility. Binary size: 113KB raw / 59KB gzipped (well under 300KB budget). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Use grouped redirects ({ cmd1; cmd2; } >> file) instead of individual
redirects to satisfy shellcheck/actionlint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Document the three I/O models evaluated (WASM-drives, host-drives, hybrid) and recommend hybrid streaming I/O for M2 with read_input and write_output host imports. Update ABI evolution to 13 functions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement the host side of the WASM crypto/IO ABI using Wazero. The host package registers "crypto" and "io" modules that fulfill the go:wasmimport calls from the hostcrypto guest package, delegating all crypto operations to lib/ocrypto. Host functions: random_bytes, aes_gcm_encrypt/decrypt, hmac_sha256, rsa_oaep_sha1_encrypt/decrypt, rsa_generate_keypair, get_last_error, read_input, write_output. Includes 20 tests covering ABI conformance (programmatic WASM module with matching imports), happy-path round-trips, error paths (bad keys, corrupted ciphertext, wrong-key decrypt, OOB writes, nil I/O), error state lifecycle, and truncation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Replace the tdf_encrypt stub with a complete single-segment TDF3 encrypt path running inside the WASM sandbox. All crypto is delegated to the host via hostcrypto; manifest construction, policy binding, HS256 integrity, and ZIP assembly run inside WASM using tinyjson types and zipstream. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 11 integration tests that compile the WASM module, load it in wazero, and exercise tdf_encrypt end-to-end: round-trip decrypt, manifest field validation, policy binding, segment/root integrity, UUID format, attributes, empty plaintext, deterministic sizes, and error paths. Fix Go 1.25 wasip1 proc_exit(0) closing the module by using wazero's FunctionExporter to provide real WASI with a custom proc_exit that panics (non-sys.ExitError) instead of closing. Fix GC-safety in malloc by keeping allocations reachable, and copy data in ptrToString/ptrToBytes instead of using raw unsafe references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…ckage Extract 7 TinyGo-compatible functions (CalculateSHA256, SHA256AsHex, CalculateSHA256Hmac, SHA256HmacAsHex, Base64Encode, Base64Decode, RandomBytes) into lib/ocrypto/cryptoutil/ so the WASM module can import them without pulling in crypto/rsa, crypto/aes, etc. Root package becomes thin wrappers — all 60+ consumers unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add configurable integrity algorithm (HS256/GMAC) for both root and segment signatures in the WASM TDF encrypt path. GMAC extracts the last 16 bytes of GCM ciphertext as the authentication tag. Also fix segment signature in experimental Writer to cover the full encrypted blob (nonce + ciphertext + tag) for SDK decrypt compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The cryptoutil split was intended for WASM direct imports, but all crypto must stay host-delegated for FIPS pluggability. Restore the original implementations in crypto_utils.go and delete the unused sub-package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
X-Test Failure Report |
Document that all crypto must stay host-delegated to preserve FIPS backend pluggability at deployment time. No crypto packages should be compiled into the WASM binary. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
TinyGo's wasi-libc already exports `malloc` and `free`, causing duplicate export errors during wasm-opt. Prefixing with `tdf_` avoids the conflict and enables TinyGo builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
…ment TDFs The zipstream writer unconditionally used data descriptors (bit 3) with zero-valued CRC/sizes in the local file header, even though encrypt.go computes the full ciphertext and CRC32 before calling WriteSegment. STORED entries with data descriptors are rejected by Java's ZipInputStream and other strict ZIP parsers. For single-segment TDFs the total payload size equals segment 0's size, so the values can go directly in the header. Multi-segment TDFs still use data descriptors since the total isn't known when segment 0's header is written. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
… spike Update the spike doc to reflect that the same tdfcore.wasm binary has been validated on all three SDK target runtimes: Go (Wazero), browser (SubtleCrypto), and JVM (Chicory). Add cross-platform validation results section and update the architecture diagram. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Implement decrypt capability in the WASM TDF module to complete the read/write round-trip within the sandbox. Uses the existing host ABI (aes_gcm_decrypt, hmac_sha256) with no new host functions required. - decrypt.go: ZIP parser (forward-scan, TinyGo-safe), manifest unmarshaling via tinyjson, segment + root integrity verification, AES-GCM decryption via hostcrypto - main.go: tdf_decrypt wasmexport (tdfData + DEK in, plaintext out) - decrypt_test.go: 7 integration tests covering round-trip, all 4 algorithm combos, empty payload, integrity failure, wrong DEK, invalid ZIP, and buffer-too-small Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…oss-SDK tests Fix segment integrity signature computation in the WASM TDF module to include the GCM nonce prefix (matching the standard SDK). The WASM module was computing signatures over ciphertext+tag only, causing cross-SDK incompatibility. Add multi-segment encrypt/decrypt support, ZIP64 parsing, and cross-SDK verification tests using both the experimental Writer and wazero-based WASM decrypt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… and WASM Add benchmark-cross-sdk command that measures encrypt/decrypt performance across all three TDF implementations with configurable payload sizes and iterations. Uses 1MB Writer segments for optimal parallel throughput. Also adds encrypt() method to wasmRuntime and fixes decrypt() output buffer to scale with TDF size instead of hardcoded 1MB cap. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…opies Replace ptrToBytes with zero-copy ptrToSlice for TDF input and DEK, write decrypted segments directly into the host-provided output buffer via new AesGcmDecryptInto, removing ~3x payload memory overhead that caused OOM at 100MB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Summary
sdk/experimental/tdf/wasm/that exercise the stdlib and third-party packages needed for a hybrid WASM TDF core enginesdk/experimental/tdf/manifest, assertion, and key-access structs fromencoding/jsonto tinyjson codegen for TinyGo WASM compatibility — includes generated marshal/unmarshal code and updated teststinyjsonandzipstreamcanaries that validate these migrations compile and round-trip correctly under TinyGo.github/workflows/tinygo-wasm-canary.yaml) that compiles each canary with TinyGo targetingwasip1— expected to have failures until spike work is completedocs/adr/spike-wasm-core-tinygo-hybrid.md) documenting the TinyGo hybrid architecture, host crypto ABI (8 functions), go/no-go criteria, and task breakdownsdk/benchmark_test.go) measuring pure decrypt throughput at 1MB–2GB using direct key injection (no KAS dependency)sdk/experimental/tdf/benchmark_test.go) measuring streaming encrypt throughput, single-segment performance, and full TDF assemblyhostcryptopackage (sdk/experimental/tdf/wasm/hostcrypto/) with typed Go wrappers for allgo:wasmimporthost functions — the WASM-side ABI layer for crypto and I/Osdk/experimental/tdf/wasm/host/) implementing the host side of the crypto/IO ABI — registers"crypto"and"io"modules with Wazero, delegating all crypto tolib/ocrypto. Includes 20 tests covering ABI conformance, round-trip correctness, error paths, and OOB handling.hostcrypto; manifest construction, policy binding (HS256), integrity computation, and ZIP assembly run inside WASM using tinyjson types and zipstream.ptrToBytescopies with zero-copyptrToSlice, writing decrypted segments directly into the host-provided output buffer viaAesGcmDecryptInto, and removing intermediate accumulation. Peak WASM Go-heap usage during decrypt drops from ~300 MB to ~0 for a 100 MB TDF.tdf_encrypt: round-trip decrypt, manifest field validation, policy binding, segment/root integrity, UUID format, attributes, empty plaintext, deterministic sizes, and error paths. Includes wazero proc_exit fix for Go 1.25 wasip1 and GC-safety fixes for WASM malloc.examples/cmd/benchmark_cross_sdk.go) comparing Production SDK, Experimental Writer, and WASM across payload sizes from 256 B to 100 MB.Canary programs
base64hexencoding/base64,encoding/hexzipwriteencoding/binary,hash/crc32,bytes,sort,synctinyjsonzipstreamiocontextio,context,strings,strconv,fmt,errorsstdjsonencoding/jsonwith TDF manifest structs (superseded bytinyjson)wasmgo:wasmimporthost ABI +tdfpackageHost crypto ABI
random_bytesocrypto.RandomBytes(n)aes_gcm_encryptocrypto.NewAESGcm(key).Encrypt(pt)aes_gcm_decryptocrypto.NewAESGcm(key).Decrypt(ct)hmac_sha256ocrypto.CalculateSHA256Hmac(key, data)rsa_oaep_sha1_encryptocrypto.NewAsymEncryption(pem).Encrypt(pt)rsa_oaep_sha1_decryptocrypto.NewAsymDecryption(pem).Decrypt(ct)rsa_generate_keypairocrypto.NewRSAKeyPair(bits)get_last_errorread_inputcfg.Input.Read(buf)write_outputcfg.Output.Write(buf)WASM TDF encrypt (Task 3.1)
Single-segment TDF3 encrypt running entirely inside the WASM sandbox:
hostcrypto.RandomBytes(32)hostcrypto.RsaOaepSha1Encrypt(kasPub, dek)MarshalJSON()HMAC-SHA256(dek, base64Policy)→ hex → base64hostcrypto.AesGcmEncrypt(dek, plaintext)HMAC-SHA256(dek, cipher)→ base64HMAC-SHA256(dek, segmentSig)→ base64WriteSegment+FinalizeEncrypt integration tests (11 tests)
TestTDFEncryptRoundTripTestTDFEncryptManifestFieldsTestTDFEncryptPolicyBindingTestTDFEncryptSegmentIntegrityTestTDFEncryptPolicyUUIDTestTDFEncryptWithAttributesTestTDFEncryptEmptyPlaintextTestTDFEncryptDeterministicSizesTestTDFEncryptErrorInvalidKeyTestTDFEncryptErrorBufferTooSmallTestTDFEncryptGetErrorClearsAfterReadget_errorretrievalCross-SDK Benchmarks
Measured on a single dev machine (Apple Silicon M3 Max); numbers are indicative, not normative. 5 iterations per size.
Encrypt
Decrypt
*Production SDK: includes KAS rewrap network latency (~20 ms round-trip to localhost)
**WASM: includes local RSA-OAEP DEK unwrap (no network); in production the host would call KAS for rewrap
WASM decrypt memory optimization
ptrToBytes)ptrToSlice)AesGcmDecryptIntowrites to output)plaintextslicetdfDecryptMicrobenchmarks
BenchmarkDecrypt/1MB–2GBsdk/benchmark_test.goBenchmarkStreamDecrypt(100MB, 32KB reads)sdk/benchmark_test.goBenchmarkWriterEncrypt/1MB–1GBsdk/experimental/tdf/benchmark_test.goBenchmarkWriterWriteSegment(2MB)sdk/experimental/tdf/benchmark_test.goBenchmarkWriterAssemble/1MB–100MBsdk/experimental/tdf/benchmark_test.goCI workflow
ciaggregation job, does not block PRsfail-fast: false— all canaries run independentlysdk/experimental/tdf/**,sdk/internal/zipstream/**,sdk/manifest.go,lib/ocrypto/**Test plan
base64hex,zipwrite,tinyjson, andzipstreamcanaries pass TinyGo compilation and executiongo build ./sdk/experimental/tdf/wasm/...)ciaggregation jobneedslistcd sdk && go test -bench=BenchmarkDecrypt -short -run=^$ .cd sdk/experimental/tdf && go test -bench=Benchmark -short -run=^$ .go test -v ./sdk/experimental/tdf/wasm/host/(44 host ABI + integration tests)go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFEncrypt(11 end-to-end tests)go test -v ./sdk/experimental/tdf/wasm/host/ -run TestTDFDecrypt(13 end-to-end tests)cd examples && go run . benchmark-cross-sdk -e ... --sizes "256,1024,16384,65536,262144,1048576,10485760,104857600"go build ./sdk/...GOOS=wasip1 GOARCH=wasm go build ./sdk/experimental/tdf/wasm/Related: SDK-WASM-1 (Jira)
🤖 Generated with Claude Code