After discussion with @wujingyue the correct solution for this is going to require a more targeted refactor around compilation responsibility and the ownership of CompiledKernel. Since this PR is getting rather large I'm opening a separate issue for this and will follow up on this with a dedicated change.
Originally posted by @mdavis36 in #5616 (comment)