Jtr/server filesystem api updates

developer_manual/server/architecture/files.rst

@@ -5,144 +5,289 @@ Nextcloud filesystem API
 High level overview
 -------------------
 
-The Nextcloud filesystem is roughly based on the unix filesystem, consisting of multiple storages
-mounted at various locations.
+The Nextcloud filesystem provides a uniform API over multiple storage backends mounted into a single virtual filesystem.
+
+Application code mainly interacts with the filesystem through two APIs:
+
+- the newer ``IRootFolder`` / ``Node`` API
+- the legacy ``View`` API
+
+New code should prefer the ``IRootFolder`` / ``Node`` API. Internally, the two APIs still interoperate, and Node operations still rely on lower-level View-based functionality in several places.
 
 .. code-block:: text
 
          ┌──────────────────────────────────┐
-         │Code wanting to use the filesystem│
-         └─────────┬─────────────────────┬──┘
-                   │                     │
-                   │                     │
-    ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
-    ╎Filesystem    │                     │         ╎
-    ╎layer         │new                  │legacy   ╎
-    ╎              │                     │         ╎
-    ╎              ▼                     ▼         ╎
-    ╎      ┌────────┐ Partly build on  ┌─┴──────┐  ╎
-    ╎      │Node API├─────────────────►│View API│  ╎
-    ╎      └───────┬┘                  └─┬──────┘  ╎
-    ╎              │                     │         ╎
-    └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
-                   │                     │
-    ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
-    ╎Storage layer │                     │         ╎
-    ╎              ├─────────────────────┤         ╎
-    ╎              │                     │         ╎
-    ╎              ▼                     ▼         ╎
-    ╎        ┌───────┐    ┌───────┐    ┌──────┐    ╎
-    ╎        │Storage│═══>│Scanner│═══>│Cache │    ╎
-    ╎        └───────┘    └───────┘    └──────┘    ╎
-    ╎                                              ╎
-    ╎                                              ╎
-    └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
+         │ Code using the filesystem        │
+         └─────────┬───────────────────┬────┘
+                   │                   │
+                   │                   │
+    ┌╌╌╌╌╌╌╌╌╌╌╌╌╌Filesystem layer╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
+    ╎              │                   │            ╎
+    ╎          new │                   │ legacy     ╎
+    ╎              ▼                   ▼            ╎
+    ╎   ┌────────────────┐   ┌──────────────────┐   ╎
+    ╎   │IRootFolder /   │   │View API          │   ╎
+    ╎   │Node API        │   │(path based)      │   ╎
+    ╎   └────────┬───────┘   └────────┬─────────┘   ╎
+    ╎            │                    │             ╎
+    ╎            └──── compatibility ─┘             ╎
+    ╎                   hooks/events                ╎
+    ╎                                               ╎
+    ╎        mount management and path routing      ╎
+    └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┬╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
+                    │
+    ┌╌╌╌╌╌╌╌╌╌╌╌╌Storage layer and metadata services╌╌╌╌╌╌╌╌╌╌┐
+    ╎                                                         ╎
+    ╎                      ┌──────────┐                       ╎
+    ╎                      │ Storage  │                       ╎
+    ╎                      └────┬─────┘                       ╎
+    ╎           ┌───────────────┼───────────────┐             ╎
+    ╎           │               │               │             ╎
+    ╎           ▼               ▼               ▼             ╎
+    ╎   ┌────────────┐  ┌────────────┐  ┌────────────┐        ╎
+    ╎   │  Scanner   │◄─│  Watcher   │  │  Updater   │        ╎
+    ╎   └──────┬─────┘  └─────┬──────┘  └──────┬─────┘        ╎
+    ╎          │              │                │              ╎
+    ╎          │              │                ▼              ╎
+    ╎          │              │         ┌────────────┐        ╎
+    ╎          │              │         │ Propagator │        ╎
+    ╎          │              │         └──────┬─────┘        ╎
+    ╎          │              │                │              ╎
+    ╎          ▼              ▼                ▼              ╎
+    ╎   ┌──────────────────────────────────────────────┐      ╎
+    ╎   │                   Cache                      │      ╎
+    ╎   └──────────────────────────────────────────────┘      ╎
+    ╎                                                         ╎
+    ╎   Wrappers can alter storage, cache and                 ╎
+    ╎   related behavior without reimplementing               ╎
+    ╎   a backend                                             ╎
+    └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
 
 Filesystem layer
 ^^^^^^^^^^^^^^^^
 
-Any code that wants to use the filesystem has two API options to use, the new ``Node`` api and the old ``View`` api.
-New code should preferably use the ``Node`` api as it allows building systems with less overhead than the old api.
+The filesystem layer exposes a virtual filesystem assembled from multiple mounts.
+
+Node API
+""""""""
+
+The preferred API for new code is the ``IRootFolder`` / ``Node`` API, exposed through interfaces such as:
+
+- ``OCP\Files\IRootFolder``
+- ``OCP\Files\Folder``
+- ``OCP\Files\File``
+- ``OCP\Files\Node``
+
+This API provides an object-oriented view of the filesystem and is the preferred entry point for application code. Typical flows start from ``IRootFolder`` and then access user folders, files, or subfolders as ``Node`` objects.
+
+View API
+""""""""
+
+The legacy API is centered around ``OC\Files\View`` and path-based operations.
+
+It is still widely used internally and remains important for compatibility. New code should avoid introducing new dependencies on ``View`` where a Node-based alternative exists.
+
+Compatibility between APIs
+""""""""""""""""""""""""""
+
+The newer and legacy APIs are not fully separate stacks. In the current implementation, Node operations still rely on lower-level View-based functionality in several places.
+
+Nextcloud also maintains compatibility between old and new event systems. Legacy filesystem hooks are bridged to Node-level events so that existing integrations continue to work while newer code can subscribe to typed events.
+
+Mount management
+""""""""""""""""
+
+The filesystem layer also manages mounts.
+
+This includes:
+
+- registering mounts provided by core or apps
+- resolving filesystem paths to a mount point and an internal path within the mounted storage
+- exposing mount information for a path or subtree
+- caching per-user mount metadata to support efficient lookup and file resolution
+
+Mount handling is a core part of the filesystem design, not just a helper around storages. A user may see the same underlying storage through different mounts, with different visibility or permissions.
+
+Metadata services
+^^^^^^^^^^^^^^^^^
+
+Each storage is paired with a metadata cache that stores information about files and folders such as path, file id, size, mtime, etag, mimetype, permissions, and related metadata.
+
+This cache is persistent and database-backed. It is not just an in-memory optimization layer: many filesystem operations depend on it for lookup, search, move, and consistency behavior.
+
+Metadata cache maintenance is shared between several components and services:
 
-Besides the filesystem apis, this layer also manages the available mounts, containing the logic to allow apps
-to setup their mounts and translating filesystem paths into a mountpoint + "internal" path.
+Scanner
+"""""""
+
+The scanner reads metadata from a storage backend and inserts or updates entries in the cache.
+
+It is used for initial discovery and for refreshing metadata from the backend.
+
+Watcher
+"""""""
+
+The watcher checks whether files or folders may have changed outside of the current Nextcloud process, determines whether cached entries should be refreshed, triggers rescans (via the Scanner),
+and may also directly update/remove stale cache entries when necessary.
+
+Updater
+"""""""
+
+The updater reacts to changes performed through the filesystem APIs, keeps cache entries in sync with those changes, and triggers propagation of parent-folder metadata updates.
+
+Propagator
+""""""""""
+
+The propagator updates parent folder metadata, such as mtimes, etags, sizes, and sometimes other information, after changes to child entries.
+
+Together, these components keep filesystem metadata coherent for both internal and external changes.
 
 Storage layer
 ^^^^^^^^^^^^^
 
-The storage implementation handles the details of communicating with the filesystem or remote storage api
-and provide a uniform api for Nextcloud to use the storage.
+The storage layer abstracts the details of concrete backends and provides a uniform API for filesystem operations.
+
+Examples of storage backends include local filesystem storage and object storage backends.
+
+Most storage implementations inherit common behavior from shared base classes and can provide additional services such as:
 
-For each storage a metadata cache/index is maintained to allow reading metadata of the storage without having
-to talk to the (potentially) slow storage backend. The scanner is responsible for updating the cache with
-information from the storage backend.
+- cache
+- scanner
+- watcher
+- updater
+- propagator
+- locking
+- streaming and direct file access helpers
 
-Storage/Cache wrappers
-----------------------
+Wrappers
+--------
+
+To allow apps and core components to customize behavior without reimplementing every backend, Nextcloud uses wrappers.
+
+A wrapper encapsulates another storage and overrides selected behavior while forwarding all other calls to the wrapped storage.
+
+This allows behavior to be composed orthogonally to backend type. For example, a wrapper can:
+
+- restrict access to a subdirectory
+- mask permissions
+- add encryption-related behavior
+- adjust metadata handling
+
+Wrappers can be stacked, so multiple behavior changes can be combined around the same base storage.
 
-To allow apps to customize the behavior of a storage without requiring the app to implement this for every
-possible storage backend, a ``Wrapper`` system is used.
+Cache and related wrappers
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-A ``Wrapper`` encapsulates an inner storage and allows overwriting any method to customize its behavior, with
-all other methods being passed through to the inner storage.
+When a storage wrapper changes visible behavior, corresponding metadata behavior usually needs to change as well.
 
-Generally search storage wrapper has an equivalent cache wrapper encapsulating the cache of the inner storage
-to provide the same behavior modifications when reading metadata from the cache.
+For that reason, many storage wrappers are paired with corresponding cache wrappers. Depending on the feature,
+wrapper-specific behavior may also extend to watcher, propagator, or other metadata-related services.
 
-Wrappers can be layered to stack the behavior of the wrappers, for example the ``groupfolders`` app works by
-stacking a wrapper to provide access to a single folder on the root storage with a wrapper to limit the permissions
-of the storage.
+For example:
+
+- a jail wrapper must map visible paths to paths inside the wrapped storage
+- a permissions wrapper must ensure cached permissions match the restricted view
+- wrapper-specific propagator or watcher logic may be needed so metadata updates remain consistent
 
 .. code-block:: text
 
-    ┌───────────────┐      ┌────────────────────┐
-    │PermissionsMask├─────►│CachePermissionsMask│  PermissionsMask applies a mask to the permissions of a storage
-    └───────┬───────┘      └─────────┬──────────┘  to provide less-privileged access to a storage
-            │                        │
-            ▼                        ▼
-    ┌───────────────┐      ┌────────────────────┐
-    │Jail           ├─────►│CacheJail           │  Jail restricts access to a file or folder of a storage providing
-    └───────┬───────┘      └─────────┬──────────┘  a limited view into the storage (think unix chroot or bind mount)
-            │                        │
-            ▼                        ▼
-    ┌───────────────┐      ┌────────────────────┐
-    │Base Storage   ├─────►│Base Cache          │
-    └───────────────┘      └────────────────────┘
-
-Code Map
+    ┌────────────────┐      ┌──────────────────────┐
+    │PermissionsMask │─────►│CachePermissionsMask  │
+    └───────┬────────┘      └──────────────────────┘
+            │
+            ▼
+    ┌────────────────┐      ┌──────────────────────┐
+    │Jail            │─────►│CacheJail             │
+    └───────┬────────┘      ├──────────────────────┤
+            │               │JailWatcher           │
+            │               ├──────────────────────┤
+            │               │JailPropagator        │
+            ▼               └──────────────────────┘
+    ┌────────────────┐      ┌──────────────────────┐
+    │Base storage    │─────►│Cache                 │
+    └────────────────┘      └──────────────────────┘
+
+A common example is the combination of:
+
+- ``Jail`` to expose only a subtree of a storage
+- ``PermissionsMask`` to reduce the effective permissions of that view
+
+This is the basis for building restricted views over shared or app-managed storage.
+
+Code map
 --------
 
-Approximate overview of the significant filesystem code
+Approximate overview of significant filesystem code.
 
 AppData
 ^^^^^^^
 
-High level api for accessing "appdata" folders, based on the ``Node/SimpleFS`` API
+High-level API for application data directories.
+
+The ``AppData`` implementation is built on top of the Node API and exposed through the simpler ``IAppData`` / ``ISimpleFile`` interfaces.
 
 Cache
 ^^^^^
 
-- ``Cache`` implementation
-- Cache wrappers
-- Scanner and cache update logic
-- Search infrastructure
+- metadata cache implementation
+- database-backed file metadata storage
+- cache wrappers
+- scanner, watcher, updater, and propagation logic
+- search and query support over cached metadata
 
 Mount
 ^^^^^
 
-Mountpoint management and setup
+- mount point representation
+- mount registration and lookup
+- user-specific mount setup and cached mount metadata
 
 Node
 ^^^^
 
-``Node`` filesystem api implementation
+- ``IRootFolder`` / ``Node`` filesystem API implementation
+- file and folder objects
+- compatibility bridging between legacy hooks and Node events
 
 ObjectStorage
 ^^^^^^^^^^^^^
 
-Implementation of the various supported object store storage backends
+Implementation of supported object storage backends and their filesystem integration.
 
 SimpleFS
 ^^^^^^^^
 
-Simplified version of the Node api, for providing a more limited api for some filesystem bits
+Simplified filesystem API used for selected use cases such as app data access.
 
 Storage
 ^^^^^^^
 
-Implementation of various storage backends and wrappers
+- common storage base classes
+- concrete storage backends
+- storage wrappers
 
 Streams
 ^^^^^^^
 
-Various low-level php stream wrapper used in storage implementations
+Low-level PHP stream wrappers used by storage implementations.
 
 Type
 ^^^^
 
-Mimetype management and detection
+Mimetype handling and detection.
 
 View.php
 ^^^^^^^^
 
-Legacy View api
+Legacy path-based filesystem API.
+
+Guidance for new code
+---------------------
+
+For new application code:
+
+- prefer ``IRootFolder`` / ``Node`` over ``View``
+- avoid depending directly on storage-specific details unless required
+- treat the metadata cache as part of the filesystem contract, not merely a performance optimization
+- be aware that mounts and wrappers can change visible paths, permissions, and metadata behavior
+- use app data and simplified abstractions such as ``SimpleFS`` where they fit the use case