REPL dependency breaks precompilation in AWS Lambda

[chang-ph]

The v1.11.1 version of MLJModelInterface has introduced REPL as a new dependency. As AWS Lambda doesn't support pseudo-terminal, precompilation of REPL always fails with this error:

This has resulted in the inability of some libraries that rely on MLJModelInterface—specifically Turing and MCMCChains for our use case—to function within the AWS Lambda environment.

[DilumAluthge]

@ablaom We might consider moving some of these introspection tools out of MLJModelInterface and into a separate package. That would allow us to remove REPL and InteractiveUtils as dependencies of MLJModelInterface.

[ablaom]

This is tricky. I think I need a pretty good argument for dodging two core Julia libraries.

REPL was added because Base.Docs.doc stopped working after a recent Julia release, which moved Docs.doc to REPL. This is used by the MLJModelInterface.doc_header which provides boilerplate for MLJ model document strings. Removing this method would be super disruptive.

Is there a workaround that avoids REPL.Docs.doc?

Here's where it is used:

MLJModelInterface.jl/src/model_traits.jl

Line 25 in b587c6e

docstring = Base.Docs.doc(C) |> string

(this won't work in latest julia unless REPL is loaded)

[DilumAluthge]

Hmmm, if the doc_header() function can't be removed, then we might be stuck. I don't know of any easy way to get the same behavior as REPL.Docs.doc without loading REPL, except for maybe vendoring that code here (but I don't really like that idea).

[DilumAluthge]

Maybe we should open this as an upstream Julia feature request. Namely, the request is the ability to precompile REPL even if you can't open a pty.

[giordano]

@chang-ph I'm confused: why do you need to precompile REPL in the first place? Did you customise JULIA_DEPOT_PATH by any chance?

[chang-ph]

I don't know why precompiling REPL. It just happens. I am not familiar with the underlying precompilation mechanism of Julia.

And we do have a customized JULIA_DEPOT_PATH: JULIA_DEPOT_PATH=/tmp/julia. That's because AWS Lambda can give a large enough space only in /tmp directory.

[giordano]

And we do have a customized JULIA_DEPOT_PATH: JULIA_DEPOT_PATH=/tmp/julia.

That's your entire problem, you must have a trailing colon to reuse the default depot, which includes precompiled stdlibs:

JULIA_DEPOT_PATH=/tmp/julia:

See the documentation: https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_DEPOT_PATH

[chang-ph]

Thank you. I'll try this and update you on the result.

[chang-ph]

Yes. It works after I add a trailing colon to JULIA_DEPOT_PATH. Thank you again.