Implement rmdir method for HNS buckets

[Mahalaxmibejugam]

rmdir method provides an implementation for deletion of empty directories for Hierarchical Namespace (HNS) enabled buckets.

• HNS-Aware Deletion: For HNS-enabled buckets, it uses the delete_folder API to perform an atomic deletion of the specified empty folder.
• Fallback Mechanism: If a bucket is not HNS-enabled or if the path refers to the bucket itself (with no folder specified), it reverts to the GCSFileSystem _rmdir implementation.
• Error Handling: It translates GCS-specific API exceptions into standard Python errors for fsspec compatibility
  1. NotFound is raised as FileNotFoundError. NotFound would be thrown by the API if the path is a file or if it doesn't exist.
  2. FailedPrecondition (for non-empty directories) is raised as OSError.
• Cache Invalidation: On successful deletion, it invalidates the cache for the deleted path and its parent to ensure the filesystem view remains consistent.
• Placeholder Object Handling: The implementation robustly handles cases where a directory contains a zero-byte placeholder object (e.g., my-folder/). It attempts to delete this placeholder before deleting the folder itself, preventing "directory not empty" errors.
  • Note on Behavior: The placeholder deletion is a separate, preceding step. If the final delete_folder operation fails (for instance, if the directory contains other files), the placeholder object will have already been removed. This side effect is considered acceptable because placeholder objects are not functionally required for HNS buckets which contain native folder entities.
• Testing:Tests are added to validate the new rmdir logic, covering successful deletions on HNS buckets, correct error handling for non-empty or non-existent directories, and ensuring the method properly falls back to the standard behavior for non-HNS buckets

[martindurant]

Question: since HNS buckets have a true folder structure, it should be possible to ammend the directory listings cache rather than simply invalidate it when deleting, no? That could avoid unnecessary listings later on.

[ankitaluthra1]

/gcbrun

[Mahalaxmibejugam]

Question: since HNS buckets have a true folder structure, it should be possible to ammend the directory listings cache rather than simply invalidate it when deleting, no? That could avoid unnecessary listings later on.

@martindurant Invalidating the cache for a specific path was also invalidating the cache for all parent directories in the hierarchy. I have updated the logic to remove the cache entry only for the deleted directory and to update its immediate parent to remove the deleted directory's entry.

[ankitaluthra1]

/gcbrun

[ankitaluthra1]

/gcbrun

[ankitaluthra1]

/gcbrun

[Mahalaxmibejugam]

Question: since HNS buckets have a true folder structure, it should be possible to ammend the directory listings cache rather than simply invalidate it when deleting, no? That could avoid unnecessary listings later on.

@martindurant Invalidating the cache for a specific path was also invalidating the cache for all parent directories in the hierarchy. I have updated the logic to remove the cache entry only for the deleted directory and to update its immediate parent to remove the deleted directory's entry.

@martindurant Can you please review the latest changes following the cache update logic?

[martindurant]

This ought to be correct, but if some some reason the entry is duplicated in the parent, you would end up removing the wrong thing. I think a comprehension would be clearer:

Suggested change

	for i, entry in enumerate(self.dircache[parent]):
	if entry.get("name") == path:
	self.dircache[parent].pop(i)
	break
	self.dircache[parent] = [ent for ent in self.dircache[parent] if ent["name"] != path]

[Mahalaxmibejugam]

Thanks for suggestion, Initially I planned something similar to your suggestion but wanted to skip iterating over entire list if we have already removed the path from parent cache to optimize the code(might not give much performance difference though).

you would end up removing the wrong thing

Just to make sure I understand this correctly, were you referring that we will not be removing duplicate entry if it exists?

[martindurant]

No: beause you are mutating the same thing you are iterating over, in the case of there being more than one item to remove, the index i will not be correct for the second one.

[Mahalaxmibejugam]

in the case of there being more than one item to remove, the index i will not be correct for the second one.

In the original implementation, the break statement is used which exits the loop immediately after the first matching item is found and removed with .pop(i). So we won't be removing any unintended entry from the cache but this implementation won't remove any duplicates if they exist.

Changing it to list comprehension is better to make the code more robust to correctly handle potential duplicates, updated the code.

[martindurant]

Sorry, you are right: pop() would not be called moe than once. Still, this is better.

[Mahalaxmibejugam]

@martindurant @ankitaluthra1 Can we add /gcbrun to run the tests against real GCS and merge this PR if changes look good?

[martindurant]

/gcbrun

[martindurant]

(I don't know if I can actually activate it! I don't seem to have access to the run logs)

[Mahalaxmibejugam]

(I don't know if I can actually activate it! I don't seem to have access to the run logs)

You have access, tests were triggered after your /gcbrun comment. We have granted access to run the CI pipeline only to Ankita and you (specifically, anyone with either the Owner or Collaborator role).

You will be able to see the logs in the checks tab.

[ankitaluthra1]

/gcbrun