Allow to live update attributes without server restart

[BigRoy]

Description of changes

Allow to live update attributes without server restart

Technical details

Custom attributes can now be added, modified, or deleted without restarting the server. Changes take effect immediately across all server instances.

---

Problem

Custom attribute changes required a server restart because the backend built Pydantic validation models from attribute definitions once at startup and cached them indefinitely. A partial hot-reload existed for enum values on ProjectEntity only, but it was incomplete, broken for all other entity types,
nd explicitly marked in a TODO comment as not supporting horizontal scaling.

---

How It Works

When an attribute is created, modified, or deleted via the API, the server dispatches a server.attributes_updated event. This is published to Redis pub/sub, so every server instance receives it — horizontal scaling is handled. Each instance then:

1. Reloads AttributeLibrary — re-fetches all attribute rows from PostgreSQL into a temporary dict, then updates each scope list (project, folder, task, etc.) in-place using .clear() + .extend(). In-place mutation is the key mechanism: ModelSet.attributes holds a direct Python
   eference to those same list objects, so it automatically sees the new data without needing to be rebuilt.

2. Clears functools.cache entries on the three cached lookup methods (inheritable_attributes, by_name, by_name_scoped) so they recompute from the fresh data.

3. Invalidates all ModelSet Pydantic model caches — every entity type's ModelSet registered its invalidate() callback with AttributeLibrary at construction time. invalidate() sets the four cached model fields (_attrib_model, _model, _post_model, _patch_model) to None. The next
   equest to any entity endpoint triggers lazy regeneration via pydantic.create_model() from the now-updated self.attributes list.

4. Clears the aiocache entry for get_attributes() in the /api/info endpoint, so clients immediately see the updated attribute list without waiting for the 5-second TTL to expire.

---

Files Changed

File	Change
ayon_server/entities/core/attrib.py	Added reload(), reload_handler(), and invalidation callback registry
ayon_server/entities/models/__init__.py	Added ModelSet.invalidate(), auto-registers with attribute_library on construction
ayon_server/api/lifespan.py	Subscribes reload_handler to server.attributes_updated at startup
api/attributes/attributes.py	Replaces all require_server_restart() calls with EventStream.dispatch("server.attributes_updated"), removes the broken single-instance partial enum hot-reload code
ayon_server/events/default_hooks.py	Adds clear_attribute_info_cache global hook for server.attributes_updated

---

Caveats

Brief inconsistency window during reload.
Between the DB fetch completing and each scope list being repopulated, there is a tiny async window where lists are momentarily empty (after .clear(), before .extend()). Any request landing in this window would see no attributes for that scope. This is extremely unlikely in practice but worth noting or high-throughput deployments.

Pydantic model regeneration cost.
After a reload, the first request to each entity type re-runs pydantic.create_model(). This is fast (milliseconds) but happens in the request path. In a highly concurrent environment, multiple requests could each trigger regeneration simultaneously. The original lazy-init pattern has the same property, so this is not new behaviour.

Reload is eventually consistent, not instantaneous.
The server.attributes_updated event travels through Redis pub/sub and the async messaging loop. There is a small lag (typically well under a second) between the API endpoint returning and the reload completing on all instances. Requests in that window still use the old models.

functools.cache holds a reference to self.
The existing @functools.cache decorators on AttributeLibrary methods retain self in the cache key, which prevents garbage collection of the instance. Since attribute_library is a singleton this is harmless, but worth knowing if the class is ever refactored.

Stale entity data in the database.
If an attribute is deleted while entities still have values stored for it, those values remain in the DB. The regenerated Pydantic model simply ignores them. This is pre-existing behaviour, not introduced here.

Additional context

This approach may be entirely stupid and wrong. But it seemed to work for me in some dangerous initial testing.

[Copilot]

Pull request overview

Enables live reloading of custom attribute definitions (and the derived Pydantic validation models) across horizontally scaled server instances by broadcasting an update event and rebuilding in-memory attribute/model caches on receipt.

Changes:

• Dispatches a server.attributes_updated event after attribute create/update/delete operations.
• Adds AttributeLibrary.reload() plus a ModelSet invalidation callback mechanism to regenerate Pydantic models lazily on next request.
• Subscribes reload handlers and clears the /api/info attributes cache on update events.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
ayon_server/events/default_hooks.py	Adds a global hook to clear aiocache when attributes are updated.
ayon_server/entities/models/init.py	Registers ModelSet.invalidate() as a callback to drop cached generated models.
ayon_server/entities/core/attrib.py	Implements attribute reload + cache clearing + invalidation callback registry.
ayon_server/api/lifespan.py	Subscribes the attribute reload handler to server.attributes_updated at startup.
api/attributes/attributes.py	Replaces “restart required” signaling with dispatching server.attributes_updated.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

clear_attribute_info_cache calls aiocache.caches.get("default").clear(), which flushes all in-process aiocache entries (permissions, actions, metrics, etc.), not just the /api/info attributes cache. This can cause avoidable cache stampedes/extra DB work after every attribute update. Consider invalidating only the get_attributes() cached key (e.g., by giving it an explicit key/namespace and deleting that key) or using a dedicated cache alias for /api/info data.

[BigRoy]

I think that's fine for now?

[Copilot]

AttributeLibrary.reload() can run concurrently for back-to-back server.attributes_updated events (global handlers are spawned via asyncio.create_task in api/messaging.py). Because reload() awaits the DB fetch, an earlier reload can finish after a later one and overwrite newer attribute data with a stale snapshot. Please serialize reloads (e.g., an async lock created inside reload() on first use, or a coalescing/debounce mechanism) so only one reload runs at a time and the latest state always wins.

[BigRoy]

Should we handle this? :)

[martastain]

@BigRoy how did you manage to make it work?

I've created a new attribute

The event was successfuly dispatched

Then i used a patch request to update the attribute on an existing folder:

But the change was not propagated to the database (pydantic throws away undeclared attributes silently, so it looks like the model is not updated)

GraphQL models weren't updated neither:

nor it shows in openapi schema.

After server restart, it works.

[BigRoy]

My test run was to:

• Add a new attribute, save.
• Then go to overview page, start setting the values of the attribute.
• Then refresh, values persisted.
• Then restart server.
• Refresh page, values still persisted.
• Remove the attribute.
• Go to overview page, attribute was gone.
• Restart server.
• Everything still seemed fine.

Let me do some more fiddling/testing when I have some time.

[martastain]

Ah. Frontend uses a combination of operations for writing and 'allAttrib' + cached hierarchy for reading. that skips a lot of validations. if we could ensure it is the case everywhere and no one uses old graphql 'attrib' node, it would be really easy to refactor the rest. but backwards compatibility :-I

[martastain]

it would be really easy to refactor the rest.

famous last words

[BigRoy]

@martastain try this monstrosity now :')

Now patching works + attributes pop up on the GraphQL explorer and work there too, without a server restart.

Caveats

It does seem that clicking "Delete attribute" on the frontend does nothing now though 🤔 This is a frontend bug it seems, see ynput/ayon-frontend#1904

Do note that on the desktop AYON python API side - any running process would need to call:

import ayon_api

con = ayon_api.get_server_api_connection()
con.reset_attributes_schema()

Because it seems to cache the attributes schema locally - otherwise e.g. get_folders_by_path would not return the new attribute - and may hence even fail on removed existing attributes. However, this behavior is not new and behaves the same as to when the server would've restarted previously.

[MustafaJafar]

So, I gave this PR a test and I found the following:

1. when updating an attribute, e.g. adding an item to the enum list. (It works).
2. When adding OR deleting an attribute, it appears as expected in the attributes page BUT in the overview page, I can't see the new attribute and still seeing the old deleted attributed.
3. When refreshing the page, it tells me to restart the server.