Minimize Calls of getRefObj/getLinkObj by drfho · Pull Request #459 · zms-publishing/ZMS

drfho · 2026-02-07T17:41:38Z

Ref:

The function getLinkObj is called very often to process a link with in zmslinkelement.py. To minimize these calls (which hits the catalog) a cache for getRefObj() in the request buffer is introduced.

Tracing the calls back it reveales that getRefObj() in zmslinkelement.py:195 will be called 18 times within ZMSLinkElement (from getObjProperty, getType, isActive, isVisible, initObjChildren, getPageExt, renderShort, etc.), and each time it resolved the link from scratch via getLinkObj().

The fix adds request-buffering to getRefObj() using the same fetchReqBuff/storeReqBuff pattern already used by getInternalLinkDict. The cache key is 'getRefObj.%s' % self.get_uid(), so each ZMSLinkElement resolves its reference object once per HTTP request instead of 12-18 times.

drfho · 2026-02-07T17:59:44Z

Hint: the debug code (to be removed) shows how the calls are reduced:

ZMS/Products/zms/_zreferableitem.py

Lines 537 to 545 in e8eb17b

    
           # DEBUG: logging/counting getLinkObj calls 
        
           request.set('getLinkObj_counter', request.get('getLinkObj_counter', 0) + 1) 
        
           standard.writeStdout(self, '[getLinkObj] url=%s, ob=%s, ref_params=%s, counter=%d'%( 
        
             url,  
        
             ob.absolute_url() if ob is not None else None,  
        
             ref_params,  
        
             request.get('getLinkObj_counter', 0) 
        
             ) 
        
           )

drfho · 2026-02-08T09:10:26Z

Products/zms/_zreferableitem.py

-        request.get('getLinkObj_counter', 0)
+        ob.id if ob is not None else None, 
+        ref_params,
+        '\n\t'.join(traceback_stack)


@zmsdev: the traceback stack now shows the last 3 function calls ending with the call of getLinkObj(). The example shows 8 Linkobjects that induce each about 8 function calls for creating the rendering. 51 of 66 functions calls are delivered from the new request-buffer.

drfho · 2026-02-09T13:59:31Z

Test Custom-Linkcontainer using getLinkObj() for rendering

@zmsdev: a typical user-test scenario reveals that a linkcontainer of e.g. 8 custom-linkelements[*] calls 52x getLinkObj() without the buffering-optimisation in the function getRefObj() (this PR) including 9x of an aquired link-element (indirect functions calls from ouside the current context and thus without any rendering, see following picture). After introducing the buffering the getLinkObj()-calls are reduced to 33 (including 1x for aquired link-element, see no. 11 in traceback.log).
So, besides the repetitive calls of getLinkObj/getRefObj() the number of aquired link-elements may create an unexpected redundancy additionally.
At this state the optimisation reduces direct-calls from 43 to 32 (about 25%)

[*] custom-linkcontainer

Test Standard-ZMSLinkContainer

If we test the rendering of 8 standard ZMSLinkElements aggregated by the standard ZMSLinkContainer, we will get 16+1 calls of getLinkObj, means 2x each Link plus 1x for aquired. Without optimisation the standard_html creates 74+1 calls (80% less redundancy).

[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:00bcad5a-bce7-4b83-948d-fc5bead41d67, Target-ID=e5 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0ba6aba3-0217-4d94-8d2f-df21bcc766b6, Target-ID=e12 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:0dd5fb6f-bfb4-4307-96f8-6add09df52a6, Target-ID=e60 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:44b58809-bfe4-4759-b86b-c30ff5ae6ae9, Target-ID=e57 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:4c17e3a2-fb35-4b5e-99cc-67bfe44347e4, Target-ID=e65 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:664eedeb-3fff-470e-91ea-d1e5036e3834, Target-ID=e138 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:93bb8004-aad3-41cb-aea6-b72d986c3e8f, Target-ID=e22 (ZMSFolder), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}
[ZMSLinkElement:getLinkObj] uid:9c704923-ca1d-4c3d-9045-07d4adddb706, Target-ID=e63 (ZMSDocument), ref_params={}

[ZMSLinkElement:getLinkObj] uid:bdafd639-6fc2-45e2-a96f-bcb66ddf7da7, Target-ID=e132 (ZMSFlexbox ACQUIRED), ref_params={}

Ref: 934f173

Ref: #459

…s-publishing/ZMS into fix_redundant_link_processing

Ref: 1. #459 2. ca3b922 3. idasm-unibe-ch/unibe-cms#1124

drfho · 2026-03-27T15:34:25Z

Alternatives to Request-Buffering with ZMS

To save Zope session data that is accessible across all users (a "shared" session or global buffer), there several options depending on the required persistence and performance.

Since we want store link data dictionaries to speed up resolution (moving beyond the request-bound ReqBuff), there are the following approaches:

1. `RAMCacheManager` (Recommended for High Performance)

Zope's built-in RAMCacheManager is designed exactly for this. It stores data in memory across all threads and users. It is much faster than the ZODB because it avoids database commits and conflict errors.

Access: You can find it in the ZMI (usually at the Zope root or inside your ZMS instance).

Usage in Python:

cache_manager = context.get_cache_manager('my_ram_cache')
cache = cache_manager.ZCacheManager_getCache()

# Store data
cache.ZCache_set(context, link_dict, view_name='link_resolution', keywords={'uid': uid})

# Fetch data
value = cache.ZCache_get(context, view_name='link_resolution', keywords={'uid': uid})

2. `Products.mcdutils` (Memcached - for Distributed Clusters)

If Products.mcdutils (Memcache) is used and the Zope setup involves multiple physical servers (a cluster), RAMCacheManager won't work because it's local to one process.

Suggestion: Use the MemCacheZCacheManager (seen in zcache.py) which provides the same API as RAMCacheManager but stores data in a shared memcached daemon.

3. ZMS-Specific Persistence (`ZMSIndex`)

If the "link data" involves just mapping UIDs to paths, ZMS already has a central facility for this: the ZMSIndex.

zmsindex = context.getZMSIndex()
catalog = zmsindex.get_catalog()
brains = catalog(get_uid=my_uid)

But ZMSIndex has a rigid minimal schema and cannot store further data about visibility etc. by default.

4. Volatile Attributes `_v_` (Thread-Shared, Non-Persistent)

If you just want a simple in-memory cache on a singleton object (like the ZMS root) that doesn't need to survive a restart:

Logic: Attributes starting with _v_ are not saved to the database. They stay in memory as long as the object stays in the ZODB cache.
```
if not hasattr(zms_root, '_v_link_cache'):
    zms_root._v_link_cache = {}
zms_root._v_link_cache[uid] = link_data
```

Comparison for your use case:

Method	Shared?	Persistent?	Best for...
ReqBuff	No	No	Single Request performance
RAMCache	Yes	No (restarts)	High-speed link resolution lookup
Memcache	Yes	No (restarts)	Multi-server/clustered Zope instances
ZMSIndex	Yes	Yes	Standard ZMS object/path lookups

minimize calls of getRefObj/getLinkObj

6e8e05e

drfho requested a review from zmsdev February 7, 2026 17:42

drfho added 2 commits February 8, 2026 02:09

debug code for call-counting

e8eb17b

added traceback for debugging

7828500

drfho commented Feb 8, 2026

View reviewed changes

more traceback for debugging

61f5064

drfho added 2 commits February 9, 2026 23:03

revoke inline-validation in zmi

e1201ac

removed debug output for merge

934f173

drfho mentioned this pull request Feb 9, 2026

reduce calls of getRefObj/getLinkObj (PR#459) #460

Merged

drfho and others added 7 commits February 10, 2026 01:00

Merge branch 'main' into fix_redundant_link_processing

f94c041

restored lines for debug output

44aff6f

Ref: 934f173

call getRefObj for embedded links

9481056

call getRefObj for embedded links

f1b0167

debug-helper: added buffer-counter for getLinkObj

307580e

Persist embed_type (attr_type)

f6f97fa

monotonized indents of former commit

b87103b

drfho added a commit that referenced this pull request Feb 11, 2026

Optimize processing ZMSLinkElement embeddings

d660a00

Ref: #459

zmsdev and others added 7 commits February 12, 2026 14:06

fixed redundant call of getLinkObj

9d69e49

Merge branch 'fix_redundant_link_processing' of https://github.com/zm…

25b6b56

…s-publishing/ZMS into fix_redundant_link_processing

Merge branch 'main' into fix_redundant_link_processing

409438b

Merge branch 'main' into fix_redundant_link_processing

d48572e

modified validateLinkObj description

b880fd5

embed remote objects and prepare impoved validation of ref-objects

7d6faad

prepare for merge

f8f06d1

drfho added a commit that referenced this pull request Feb 18, 2026

Fix redundant link processing 3 (#459)

f9c5bab

drfho added 3 commits February 18, 2026 09:56

restore debug code

bef6f77

refact conditionioning link-validation

ca3b922

Merge branch 'main' into fix_redundant_link_processing

51ff4b4

drfho added a commit that referenced this pull request Feb 18, 2026

changed linkvalidation-default on property-processing to false

dec9c05

Ref: 1. #459 2. ca3b922 3. idasm-unibe-ch/unibe-cms#1124

Merge branch 'main' into fix_redundant_link_processing

a6bfe6a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minimize Calls of getRefObj/getLinkObj#459

Minimize Calls of getRefObj/getLinkObj#459
drfho wants to merge 24 commits intomainfrom
fix_redundant_link_processing

drfho commented Feb 7, 2026 •

edited

Loading

Uh oh!

drfho commented Feb 7, 2026 •

edited

Loading

Uh oh!

drfho Feb 8, 2026 •

edited

Loading

Uh oh!

drfho commented Feb 9, 2026 •

edited

Loading

Uh oh!

drfho commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

drfho commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drfho commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drfho Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drfho commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Custom-Linkcontainer using getLinkObj() for rendering

Test Standard-ZMSLinkContainer

Uh oh!

drfho commented Mar 27, 2026

Alternatives to Request-Buffering with ZMS

1. RAMCacheManager (Recommended for High Performance)

2. Products.mcdutils (Memcached - for Distributed Clusters)

3. ZMS-Specific Persistence (ZMSIndex)

4. Volatile Attributes _v_ (Thread-Shared, Non-Persistent)

Comparison for your use case:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

drfho commented Feb 7, 2026 •

edited

Loading

drfho commented Feb 7, 2026 •

edited

Loading

drfho Feb 8, 2026 •

edited

Loading

drfho commented Feb 9, 2026 •

edited

Loading

1. `RAMCacheManager` (Recommended for High Performance)

2. `Products.mcdutils` (Memcached - for Distributed Clusters)

3. ZMS-Specific Persistence (`ZMSIndex`)

4. Volatile Attributes `_v_` (Thread-Shared, Non-Persistent)