Android: memory-map native modules from the APK (drop fake-.so / useLegacyPackaging)#210
Merged
Conversation
Faithful, behavior-preserving port of the per-ABI task graph (gradle-download-task, tarTree/untar, copyOpt, zipSitePackages, dart-bridge download+rename, packaging, abiFilters) to build.gradle.kts. Verified green: run_example builds and runs on the arm64 emulator with identical jniLibs and Python output (numpy works). Prerequisite for the native-split tasks.
sys.meta_path finder that resolves relocated CPython extension modules from their .soref markers (jniLibs lib loaded by basename). Reads markers via frozen zipimport.get_data (zip-resident) or open() (extracted), imports only builtin/frozen machinery so it runs before any native is resolvable. Host-tested: zip + extracted-dir probes, idempotent install, pure imports fall through. Not yet wired into the build.
…hase A/B) The Kotlin split tasks replace zipSitePackages: they relocate every tagged CPython extension .so (stdlib lib-dynload + site-packages) to jniLibs/<abi>/lib<mangled>.so (dotted name, '.'->'-', readable & injective), leaving a .soref marker (content = lib name) at the module's path in ABI-common stored zips. stdlib bundle is cracked post-untar; libpythonbundle.so/libpythonsitepackages.so no longer ship. Pure code -> stdlib.zip/sitepackages.zip (assets, noCompress); allowlisted packages (SERIOUS_PYTHON_ANDROID_EXTRACT_PACKAGES) -> extract.zip, excluded from sitepackages.zip. _sp_bootstrap.py embedded at stdlib.zip root. Verified by artifact inspection: 53 stdlib + 19 numpy natives mangled in jniLibs, markers present with matching libs, zips stored, no fake-zip .so. Runtime wiring (D/F) pending, so not yet runnable end-to-end.
- AndroidPlugin.java: getFilesDir/extractAsset/unzipAsset (AssetManager + java.util.zip) to copy the stored zips to disk and unpack extract.zip. - serious_python_android.dart run(): copy stdlib.zip/sitepackages.zip once (version-keyed), unpack extract.zip; sys.path = [modulePaths, programDir, extract, sitepackages.zip, stdlib.zip]; PYTHONHOME=base. No fake-zip extraction. - _sp_bootstrap.py: resolve soname to an absolute origin under nativeLibraryDir when present (legacy packaging; CPython prepends ./ to a no-slash origin which breaks bare-soname load). - build.gradle.kts: inject interim sitecustomize.py (installs finder during site); declare split jniLibs/assets outputs + always-run so AGP's native-libs merge re-packages incrementally (no flutter clean needed). Verified on arm64 emulator: run_example runs fully — bz2, sqlite, and numpy all work, with pure code from stored zips (zipimport) and ALL native modules loaded via the finder from jniLibs. Legacy packaging still on (natives extracted); dropping it for mmap is next.
…hieved) The finder now loads extension modules via Bionic's APK zip-path (base.apk!/lib/<abi>/<soname>) when libs aren't extracted (modern packaging). AndroidPlugin exports ANDROID_APK_NATIVE_PREFIX (sourceDir + abi); _sp_bootstrap prefers an extracted nativeLibraryDir copy (legacy) else the APK zip-path. The run_example app sets useLegacyPackaging=false. Proven on arm64 emulator: program runs fully (bz2, sqlite, numpy), ZERO native extraction (no .so in app storage or nativeLibraryDir), and /proc/<pid>/maps shows 32 r-xp executable mappings backed by base.apk — i.e. native modules are mmap'd directly from the APK. Pure code loads from stored zips via zipimport.
…l via dart-bridge shim (F) - copyDartBridge_$abi uses a local dir of cross-compiled libdart_bridge-android-<abi>-py<ver>.so when SERIOUS_PYTHON_DART_BRIDGE_DIST is set (mirrors SERIOUS_PYTHON_BUILD_DIST), bypassing the GitHub download for dev iteration on the bridge. - Drop the interim sitecustomize injection: the dart-bridge Android shim now installs the finder before site (proven on emulator: green with finder installed solely by the shim). Re-enable the sitecustomize fallback for bridges without the shim (commented in the task). This branch now requires the dart-bridge android-native-mmap bridge (via the override until released).
- _sp_bootstrap.install(): report any extension module already imported at install time (loaded during core-init before the finder) — verified empty on emulator, so PyConfig core-init pulls only builtin/frozen modules (F's pre-site install is safe). - cleanStaleAbis: remove jniLibs/<abi> dirs for ABIs not in the current set (e.g. stale armeabi-v7a 3.12 leftovers when building 3.14) so old fake-zip .so aren't packaged. Audits on arm64 emulator + AAB: importtime (no pre-finder natives), ABI-split (per-ABI natives, ~33MB pure payload shared ONCE in assets vs per-ABI duplication before), multi-ABI markers resolve in x86_64, allowlist partition+extract+import, 16KB-aligned.
dart run serious_python:gen_version_tables --release-date 20260614 — the 20260614 manifest now pins dart_bridge_version 1.4.0 (the full-API/PyConfig Android bridge that installs the native-module finder before site). Python versions unchanged. Default build now downloads the released v1.4.0 bridge; verified green on the arm64 emulator (sqlite, numpy) with no env overrides. SERIOUS_PYTHON_DART_BRIDGE_DIST remains as a dev escape hatch.
…mmap (CHANGELOGs) - run_example/flask_example READMEs: remove the useLegacyPackaging/keepDebugSymbols packaging block — consumers need no special native packaging now, just minSdk 23+. Document SERIOUS_PYTHON_ANDROID_EXTRACT_PACKAGES for path-hungry packages. - run_example app build.gradle.kts: drop the packaging block entirely (modern packaging is AGP's default at minSdk 23+). Verified green on emulator with natives still mmap'd from base.apk (32 r-xp maps), zero extraction. - serious_python + serious_python_android CHANGELOGs (3.0.0): replace the useLegacyPackaging language with the native-mmap design; bump bundled dart_bridge to 1.4.0.
…builds) The Flask app binds a local socket server, so it needs android.permission.INTERNET in all build types. It was only in the debug/profile manifests, so release builds hit 'PermissionError: [Errno 1] Operation not permitted' on socket() (the app isn't in the inet group). Verified: release APK now grants INTERNET and the Flask server starts (Running on http://127.0.0.1:55001) — flask/werkzeug from the zip, _socket via the finder.
…etNativeLibraryDir - ANDROID_APK_NATIVE_PREFIX now points at whichever installed APK actually contains lib/<abi>/ — base.apk for single-APK builds, the per-ABI config split for Play Store AAB installs (where base.apk has no native libs). Detected by probing each of sourceDir + splitSourceDirs for the always-present libdart_bridge.so. - Remove the unused getNativeLibraryDir method-channel handler (run() uses getFilesDir). Verified with bundletool: built the AAB, build-apks + install-apks the device splits (base.apk + split_config.arm64_v8a.apk), ran green (numpy). adb-root /proc/<pid>/maps shows 33 r-xp mappings from split_config.arm64_v8a.apk, 0 from base.apk, 0 extracted — natives mmap'd directly from the config split.
Document, per platform, where the stdlib and site-packages live, how native extension modules are shipped (Android: jniLibs mmap'd from the APK via a custom importer; iOS: xcframeworks + AppleFrameworkLoader; macOS: universal .so; Linux/Windows: on-disk; Web: Pyodide wheels), architecture handling, and how the user's app.zip is produced and extracted at runtime. Replace the stale Android note (enableUncompressedNativeLibs / extractNativeLibs) with the new no-config guidance (minSdk 23+).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Redesigns serious_python's Android Python packaging to mirror the iOS approach, replacing the old "fake
.sozips injniLibs+useLegacyPackaging+ runtime extraction" scheme.Old scheme: the stdlib and site-packages were zipped, renamed to fake
.sofiles, dropped intojniLibs, shipped withuseLegacyPackaging=true(so they extracted to disk at install), then unzipped again at runtime.New scheme:
jniLibs/<abi>/lib<mangled>.soand memory-mapped directly from the APK — no extraction. A customsys.meta_pathfinder resolves them via.sorefmarker files (the Android analog of iOS.fwork).zipimport— the stdlib is no longer duplicated per ABI.__file__/pkg_resourcesinstead ofimportlib.resources) are shipped extracted to disk via the newSERIOUS_PYTHON_ANDROID_EXTRACT_PACKAGESenv var.PyConfigshim (site_import=0) installs the finder beforesiteruns.Result: no
useLegacyPackaging/keepDebugSymbols/extractNativeLibs, smaller apps, modern packaging atminSdk 23+.Key pieces
_sp_bootstrap.py— thesys.meta_pathfinder (.sorefprobe viazipimport.get_data, resolves tojniLibssoname or the APK zip-path).android/build.gradle.kts— migrated to Kotlin DSL; split tasks mangle.so→jniLibs, write.sorefmarkers, build stored stdlib/sitepackages zips, and partition the extract set intoextract.zip.AndroidPlugin.java— split-awareapkNativePrefix(probessourceDir+splitSourceDirs) so natives resolve fromsplit_config.<abi>.apkunder Play Store AAB delivery, plus asset extraction helpers.serious_python_run.c— Android-specificPyConfiginit + finder install beforesite.main().Verification
Validated end-to-end on an arm64 emulator through
flet build apkandflet build aab(bundletool split install): native modules mmap'd from the APK / config split (zero extraction), pure Python imported from the stored zip, the extract set (e.g.certifi) extracted to disk, app launched. importtime audit confirms no native loads occur before the finder is installed.Depends on dart_bridge 1.4.0 (already released).