Possible race condition in shutil.rmtree() within ProtocolDAG::execute_DAG()

[mjw99]

Seeing the following with a very basic AHFE benzene example, with openfe_1.8.0-apptainer.sif under Rocky 8:

Traceback (most recent call last):
  File "/home/m.williamson/tmp/openfe_cookbook/AHFE_benzene.py", line 90, in <module>
    dag_results = execute_DAG(dag, scratch_basedir=path, shared_basedir=path, n_retries=3)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/gufe/protocols/protocoldag.py", line 473, in execute_DAG
    shutil.rmtree(shared_path)
  File "/opt/conda/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/opt/conda/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/opt/conda/lib/python3.12/shutil.py", line 662, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=dirfd)
OSError: [Errno 39] Directory not empty: PosixPath('AHFE_benzene_results_20251219_1051/shared_AbsoluteSolvationSolventUnit-1ef7da26cae14b68b39d0f95a08d3ec0_attempt_0')

The directory in question is empty. Note, with the exact same example and openfe_1.7.0-apptainer.sif, this does not happen.

The following script was used:

# https://docs.openfree.energy/en/stable/tutorials/ahfe_tutorial.html


# 1) Load molecule
# =================
from rdkit import Chem
import openfe
supp = Chem.SDMolSupplier("./input/benzene.sdf", removeHs=False)
ligands = [openfe.SmallMoleculeComponent.from_rdkit(mol) for mol in supp]

# 2) Creating ChemicalSystems
# ===========================

# defaults are water
from openff.units import unit
solvent = openfe.SolventComponent()



# In state A the ligand is fully interacting in the solvent
systemA = openfe.ChemicalSystem({
    'ligand': ligands[0],
    'solvent': solvent,
}, name=ligands[0].name)

# In state B the ligand is fully decoupled in the solvent, therefore we are only defining the solvent here
systemB = openfe.ChemicalSystem({
    'solvent': solvent})


# 3) Defining the AHFE simulation settings and creating a Protocol
# ================================================================
from openfe.protocols.openmm_afe import AbsoluteSolvationProtocol
settings = AbsoluteSolvationProtocol.default_settings()

from openff.units import unit


# By default we run 3 repeats with solvent and vacuum simulation lengths of 10 ns and 2 ns over 14 lambda windows. 
# To speed things up here we instead run 1 repeat with both solvent and vacuum simulation lengths of 0.5 ns over 14 lambda windows.


settings.solvent_engine_settings.compute_platform = 'CUDA'
settings.vacuum_engine_settings.compute_platform = 'CUDA'

print(settings)

protocol = AbsoluteSolvationProtocol(settings=settings)

# 4) Running the AHFE simulation
# ===============================
dag = protocol.create(stateA=systemA, stateB=systemB, mapping=None)

from gufe.protocols import execute_DAG
import pathlib

# Finally we can run the simulations
import datetime

x = datetime.datetime.now()
path = pathlib.Path('./AHFE_benzene_results_' + x.strftime('%Y%m%d_%H%M') )
path.mkdir()

# Execute the DAG
dag_results = execute_DAG(dag, scratch_basedir=path, shared_basedir=path, n_retries=3)


# 6) Analysis
# ============

# Get the complex and solvent results
protocol_results = protocol.gather([dag_results])

print(f"AHFE dG: {protocol_results.get_estimate()}, err {protocol_results.get_uncertainty()}")

# Save the results in a json file
import gzip
import json
import gufe
outdict = {
    "estimate": protocol_results.get_estimate(),
    "uncertainty": protocol_results.get_uncertainty(),
    "protocol_result": protocol_results.to_dict(),
    "unit_results": {
        unit.key: unit.to_keyed_dict()
        for unit in dag_results.protocol_unit_results
    }
}

with open("AHFE_benzene_results.json", "a+") as stream:
    json.dump(outdict, stream, cls=gufe.tokenization.JSON_HANDLER.encoder)

I think the issue is here and could perhaps be solved by using "ignore_errors=True" in shutil.rmtree (see here )

[atravitz]

thanks for raising this @mjw99 - we're looking into it!

[atravitz]

Hi @mjw99, I have a couple questions to help us understand what exactly is happening:

1. Is what you report seeing with the v1.8 and v1.7 apptainer builds reproducible, or is this a sporadic error?
2. Are you running these workflows directly on an NFS or similar shared storage, or are you running on scratch or a local filesystem? We would expect to see this type of issue if attempting to run directly on an NFS server.

We're hesitant to use ignore_errors=True because it could result in ignoring other errors that we actually do want to interrupt runtime. I have a PR open for an another possible solution here: OpenFreeEnergy/gufe#703.

[mjw99]

Thank you for looking at this and apologies for the delay in replying

Is what you report seeing with the v1.8 and v1.7 apptainer builds reproducible, or is this a sporadic error?

Yes, very reproducible.

Are you running these workflows directly on an NFS or similar shared storage, or are you running on scratch or a local filesystem? We would expect to see this type of issue if attempting to run directly on an NFS server.

This is running on a worker node that is writing via NFS to a head node (in both v1.8 v1.7). I will follow up with some testing with v1.8 whereby all file IO is local to the node.

[mjw99]

I will follow up with some testing with v1.8 whereby all file IO is local to the node.
Apologies for the delay in following up; it has been quite busy.

Yes, NFS was the issue; writing locally on the node resolved the issue with 1.8 . Thank you again Alyssa for looking at this and solving it with your suggestion.