diff --git a/developer_manual/server/architecture/files.rst b/developer_manual/server/architecture/files.rst index 9fb0533a128..aa704e46613 100644 --- a/developer_manual/server/architecture/files.rst +++ b/developer_manual/server/architecture/files.rst @@ -5,144 +5,289 @@ Nextcloud filesystem API High level overview ------------------- -The Nextcloud filesystem is roughly based on the unix filesystem, consisting of multiple storages -mounted at various locations. +The Nextcloud filesystem provides a uniform API over multiple storage backends mounted into a single virtual filesystem. + +Application code mainly interacts with the filesystem through two APIs: + +- the newer ``IRootFolder`` / ``Node`` API +- the legacy ``View`` API + +New code should prefer the ``IRootFolder`` / ``Node`` API. Internally, the two APIs still interoperate, and Node operations still rely on lower-level View-based functionality in several places. .. code-block:: text ┌──────────────────────────────────┐ - │Code wanting to use the filesystem│ - └─────────┬─────────────────────┬──┘ - │ │ - │ │ - ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ - ╎Filesystem │ │ ╎ - ╎layer │new │legacy ╎ - ╎ │ │ ╎ - ╎ ▼ ▼ ╎ - ╎ ┌────────┐ Partly build on ┌─┴──────┐ ╎ - ╎ │Node API├─────────────────►│View API│ ╎ - ╎ └───────┬┘ └─┬──────┘ ╎ - ╎ │ │ ╎ - └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ - │ │ - ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ - ╎Storage layer │ │ ╎ - ╎ ├─────────────────────┤ ╎ - ╎ │ │ ╎ - ╎ ▼ ▼ ╎ - ╎ ┌───────┐ ┌───────┐ ┌──────┐ ╎ - ╎ │Storage│═══>│Scanner│═══>│Cache │ ╎ - ╎ └───────┘ └───────┘ └──────┘ ╎ - ╎ ╎ - ╎ ╎ - └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ + │ Code using the filesystem │ + └─────────┬───────────────────┬────┘ + │ │ + │ │ + ┌╌╌╌╌╌╌╌╌╌╌╌╌╌Filesystem layer╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐ + ╎ │ │ ╎ + ╎ new │ │ legacy ╎ + ╎ ▼ ▼ ╎ + ╎ ┌────────────────┐ ┌──────────────────┐ ╎ + ╎ │IRootFolder / │ │View API │ ╎ + ╎ │Node API │ │(path based) │ ╎ + ╎ └────────┬───────┘ └────────┬─────────┘ ╎ + ╎ │ │ ╎ + ╎ └──── compatibility ─┘ ╎ + ╎ hooks/events ╎ + ╎ ╎ + ╎ mount management and path routing ╎ + └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┬╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ + │ + ┌╌╌╌╌╌╌╌╌╌╌╌╌Storage layer and metadata services╌╌╌╌╌╌╌╌╌╌┐ + ╎ ╎ + ╎ ┌──────────┐ ╎ + ╎ │ Storage │ ╎ + ╎ └────┬─────┘ ╎ + ╎ ┌───────────────┼───────────────┐ ╎ + ╎ │ │ │ ╎ + ╎ ▼ ▼ ▼ ╎ + ╎ ┌────────────┐ ┌────────────┐ ┌────────────┐ ╎ + ╎ │ Scanner │◄─│ Watcher │ │ Updater │ ╎ + ╎ └──────┬─────┘ └─────┬──────┘ └──────┬─────┘ ╎ + ╎ │ │ │ ╎ + ╎ │ │ ▼ ╎ + ╎ │ │ ┌────────────┐ ╎ + ╎ │ │ │ Propagator │ ╎ + ╎ │ │ └──────┬─────┘ ╎ + ╎ │ │ │ ╎ + ╎ ▼ ▼ ▼ ╎ + ╎ ┌──────────────────────────────────────────────┐ ╎ + ╎ │ Cache │ ╎ + ╎ └──────────────────────────────────────────────┘ ╎ + ╎ ╎ + ╎ Wrappers can alter storage, cache and ╎ + ╎ related behavior without reimplementing ╎ + ╎ a backend ╎ + └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘ Filesystem layer ^^^^^^^^^^^^^^^^ -Any code that wants to use the filesystem has two API options to use, the new ``Node`` api and the old ``View`` api. -New code should preferably use the ``Node`` api as it allows building systems with less overhead than the old api. +The filesystem layer exposes a virtual filesystem assembled from multiple mounts. + +Node API +"""""""" + +The preferred API for new code is the ``IRootFolder`` / ``Node`` API, exposed through interfaces such as: + +- ``OCP\Files\IRootFolder`` +- ``OCP\Files\Folder`` +- ``OCP\Files\File`` +- ``OCP\Files\Node`` + +This API provides an object-oriented view of the filesystem and is the preferred entry point for application code. Typical flows start from ``IRootFolder`` and then access user folders, files, or subfolders as ``Node`` objects. + +View API +"""""""" + +The legacy API is centered around ``OC\Files\View`` and path-based operations. + +It is still widely used internally and remains important for compatibility. New code should avoid introducing new dependencies on ``View`` where a Node-based alternative exists. + +Compatibility between APIs +"""""""""""""""""""""""""" + +The newer and legacy APIs are not fully separate stacks. In the current implementation, Node operations still rely on lower-level View-based functionality in several places. + +Nextcloud also maintains compatibility between old and new event systems. Legacy filesystem hooks are bridged to Node-level events so that existing integrations continue to work while newer code can subscribe to typed events. + +Mount management +"""""""""""""""" + +The filesystem layer also manages mounts. + +This includes: + +- registering mounts provided by core or apps +- resolving filesystem paths to a mount point and an internal path within the mounted storage +- exposing mount information for a path or subtree +- caching per-user mount metadata to support efficient lookup and file resolution + +Mount handling is a core part of the filesystem design, not just a helper around storages. A user may see the same underlying storage through different mounts, with different visibility or permissions. + +Metadata services +^^^^^^^^^^^^^^^^^ + +Each storage is paired with a metadata cache that stores information about files and folders such as path, file id, size, mtime, etag, mimetype, permissions, and related metadata. + +This cache is persistent and database-backed. It is not just an in-memory optimization layer: many filesystem operations depend on it for lookup, search, move, and consistency behavior. + +Metadata cache maintenance is shared between several components and services: -Besides the filesystem apis, this layer also manages the available mounts, containing the logic to allow apps -to setup their mounts and translating filesystem paths into a mountpoint + "internal" path. +Scanner +""""""" + +The scanner reads metadata from a storage backend and inserts or updates entries in the cache. + +It is used for initial discovery and for refreshing metadata from the backend. + +Watcher +""""""" + +The watcher checks whether files or folders may have changed outside of the current Nextcloud process, determines whether cached entries should be refreshed, triggers rescans (via the Scanner), +and may also directly update/remove stale cache entries when necessary. + +Updater +""""""" + +The updater reacts to changes performed through the filesystem APIs, keeps cache entries in sync with those changes, and triggers propagation of parent-folder metadata updates. + +Propagator +"""""""""" + +The propagator updates parent folder metadata, such as mtimes, etags, sizes, and sometimes other information, after changes to child entries. + +Together, these components keep filesystem metadata coherent for both internal and external changes. Storage layer ^^^^^^^^^^^^^ -The storage implementation handles the details of communicating with the filesystem or remote storage api -and provide a uniform api for Nextcloud to use the storage. +The storage layer abstracts the details of concrete backends and provides a uniform API for filesystem operations. + +Examples of storage backends include local filesystem storage and object storage backends. + +Most storage implementations inherit common behavior from shared base classes and can provide additional services such as: -For each storage a metadata cache/index is maintained to allow reading metadata of the storage without having -to talk to the (potentially) slow storage backend. The scanner is responsible for updating the cache with -information from the storage backend. +- cache +- scanner +- watcher +- updater +- propagator +- locking +- streaming and direct file access helpers -Storage/Cache wrappers ----------------------- +Wrappers +-------- + +To allow apps and core components to customize behavior without reimplementing every backend, Nextcloud uses wrappers. + +A wrapper encapsulates another storage and overrides selected behavior while forwarding all other calls to the wrapped storage. + +This allows behavior to be composed orthogonally to backend type. For example, a wrapper can: + +- restrict access to a subdirectory +- mask permissions +- add encryption-related behavior +- adjust metadata handling + +Wrappers can be stacked, so multiple behavior changes can be combined around the same base storage. -To allow apps to customize the behavior of a storage without requiring the app to implement this for every -possible storage backend, a ``Wrapper`` system is used. +Cache and related wrappers +^^^^^^^^^^^^^^^^^^^^^^^^^^ -A ``Wrapper`` encapsulates an inner storage and allows overwriting any method to customize its behavior, with -all other methods being passed through to the inner storage. +When a storage wrapper changes visible behavior, corresponding metadata behavior usually needs to change as well. -Generally search storage wrapper has an equivalent cache wrapper encapsulating the cache of the inner storage -to provide the same behavior modifications when reading metadata from the cache. +For that reason, many storage wrappers are paired with corresponding cache wrappers. Depending on the feature, +wrapper-specific behavior may also extend to watcher, propagator, or other metadata-related services. -Wrappers can be layered to stack the behavior of the wrappers, for example the ``groupfolders`` app works by -stacking a wrapper to provide access to a single folder on the root storage with a wrapper to limit the permissions -of the storage. +For example: + +- a jail wrapper must map visible paths to paths inside the wrapped storage +- a permissions wrapper must ensure cached permissions match the restricted view +- wrapper-specific propagator or watcher logic may be needed so metadata updates remain consistent .. code-block:: text - ┌───────────────┐ ┌────────────────────┐ - │PermissionsMask├─────►│CachePermissionsMask│ PermissionsMask applies a mask to the permissions of a storage - └───────┬───────┘ └─────────┬──────────┘ to provide less-privileged access to a storage - │ │ - ▼ ▼ - ┌───────────────┐ ┌────────────────────┐ - │Jail ├─────►│CacheJail │ Jail restricts access to a file or folder of a storage providing - └───────┬───────┘ └─────────┬──────────┘ a limited view into the storage (think unix chroot or bind mount) - │ │ - ▼ ▼ - ┌───────────────┐ ┌────────────────────┐ - │Base Storage ├─────►│Base Cache │ - └───────────────┘ └────────────────────┘ - -Code Map + ┌────────────────┐ ┌──────────────────────┐ + │PermissionsMask │─────►│CachePermissionsMask │ + └───────┬────────┘ └──────────────────────┘ + │ + ▼ + ┌────────────────┐ ┌──────────────────────┐ + │Jail │─────►│CacheJail │ + └───────┬────────┘ ├──────────────────────┤ + │ │JailWatcher │ + │ ├──────────────────────┤ + │ │JailPropagator │ + ▼ └──────────────────────┘ + ┌────────────────┐ ┌──────────────────────┐ + │Base storage │─────►│Cache │ + └────────────────┘ └──────────────────────┘ + +A common example is the combination of: + +- ``Jail`` to expose only a subtree of a storage +- ``PermissionsMask`` to reduce the effective permissions of that view + +This is the basis for building restricted views over shared or app-managed storage. + +Code map -------- -Approximate overview of the significant filesystem code +Approximate overview of significant filesystem code. AppData ^^^^^^^ -High level api for accessing "appdata" folders, based on the ``Node/SimpleFS`` API +High-level API for application data directories. + +The ``AppData`` implementation is built on top of the Node API and exposed through the simpler ``IAppData`` / ``ISimpleFile`` interfaces. Cache ^^^^^ -- ``Cache`` implementation -- Cache wrappers -- Scanner and cache update logic -- Search infrastructure +- metadata cache implementation +- database-backed file metadata storage +- cache wrappers +- scanner, watcher, updater, and propagation logic +- search and query support over cached metadata Mount ^^^^^ -Mountpoint management and setup +- mount point representation +- mount registration and lookup +- user-specific mount setup and cached mount metadata Node ^^^^ -``Node`` filesystem api implementation +- ``IRootFolder`` / ``Node`` filesystem API implementation +- file and folder objects +- compatibility bridging between legacy hooks and Node events ObjectStorage ^^^^^^^^^^^^^ -Implementation of the various supported object store storage backends +Implementation of supported object storage backends and their filesystem integration. SimpleFS ^^^^^^^^ -Simplified version of the Node api, for providing a more limited api for some filesystem bits +Simplified filesystem API used for selected use cases such as app data access. Storage ^^^^^^^ -Implementation of various storage backends and wrappers +- common storage base classes +- concrete storage backends +- storage wrappers Streams ^^^^^^^ -Various low-level php stream wrapper used in storage implementations +Low-level PHP stream wrappers used by storage implementations. Type ^^^^ -Mimetype management and detection +Mimetype handling and detection. View.php ^^^^^^^^ -Legacy View api +Legacy path-based filesystem API. + +Guidance for new code +--------------------- + +For new application code: + +- prefer ``IRootFolder`` / ``Node`` over ``View`` +- avoid depending directly on storage-specific details unless required +- treat the metadata cache as part of the filesystem contract, not merely a performance optimization +- be aware that mounts and wrappers can change visible paths, permissions, and metadata behavior +- use app data and simplified abstractions such as ``SimpleFS`` where they fit the use case