Dart now has support of build hooks and perhaps these could be used to serve precompiled binaries for desktop platforms, so additional manual steps to build and include them into the project won't be necessary.
Several other plugins been doing that even before build hooks been introduced. E.g. https://pub.dev/packages/flutter_soloud which provides native libraries for all supported platforms out of the box.