Central spider cache with optional module paths

[Micket]

I provide a "test" tree for modules that are not yet ready, perhaps need testing by some users to verify that it's ok before proceeding.

For this purpose i have a /apps/test vs /apps/prod. In each tree, I provide a central spider cache via

/apps/test/modules/cacheDir/...
/apps/prod/modules/cacheDir/...

Users have to opt in to using the test tree via module use /apps/test/modules/all.

I don't see any way of providing spider cache for both of these. or, even either one of them.

$ cat /etc/lmodrc.lua 
scDescriptT =  {
   {
      ["dir"]       = "/apps/test/modules/cacheDir/",
      ["timestamp"] = "/apps/test/modules/cacheTS.txt",
   },
   {
      ["dir"]       = "/apps/prod/modules/cacheDir/",
      ["timestamp"] = "/apps/prod/modules/cacheTS.txt",
   },

won't work as it just concatenates both caches. module spider isn't filtering on MODULEPATH at all.

I currently opted for a /etc/lmodrc.lua that just conditionally checks for each subdirectory

local function appendCacheForModulePath(mp)
   local cacheDir = mp .. "/../cacheDir"
   local timestampFile = mp .. "/../cacheTS.txt"

   table.insert(scDescriptT, {
      dir = cacheDir,
      timestamp = timestampFile,
   })
end

local modulePathList = os.getenv("MODULEPATH")
scDescriptT =  {}
if modulePathList then
   for mp in modulePathList:gmatch("[^:]+") do
      print(string.format("echo mp = %s", mp))
      appendCacheForModulePath(mp)
   end
end

but I'm also not sure if i simply reinventing some feature that already existed and i just missed.

[mrcawood]

Hi there, you can now enable dynamic spider cache selection by adding the following to your site's lmodrc.lua file:

filter_spider_cache_by_modulepath = true

When this option is enabled, Lmod will only use the system spider caches that correspond to directories in your active MODULEPATH. This will solve the issue you described by keeping your 'prod' and 'test' environments separate for module spider, depending on which module paths you are currently using.

Please use the IS783-cache_select branch to test this out, and report back.