Slow and strange behaviour when using KVM model; possibly due to `compute_ancillaries`

[PinkShnack]

Reporting some strange behaviour with Pyjibe when using the (extension) KVM model.

Issues:

• First, the calculation is slow, and gets progressively slower as Pyjibe goes through the folder after clicking "Apply Model and Fit All". However, going to a new folder speeds up again and the slowing down "starts again"
• After running "Apply Model and Fit All", the displayed rating sometimes changes when the user clicks on a single dataset in the file list.

Questions for understanding:

1. Is the rating system deterministic?
2. Is it correct that the ancillary parameter calculation get progressively slower as pyjibe runs through the files? Is this due to the curves needing the previous curve calculations because they are one dataset? Or is it just a bug?
3. The ancillary parameters seems to bounce between very small values and the values they land on, presumably because they are being fit between an interval.

Possible reasons/solutions

I added caching to nanite/model/core/compute_ancillaries, which makes the calculation fast. However, max_indent ends up being nan which of course changes the expected ancillary param values. You even wrote a TODO note in this function for caching, but I am not yet familiar enough to implement it. Any ideas would be appreciated:

    def compute_ancillaries(self, fd):
        # TODO:
        # - ancillaries are not cached yet (some ancillaries might depend on
        #   fitting interval or other initial parameters - take that into
        #   account)
        # - "max_indent" actually belongs to "common_ancillaries" (see fit.py)"

[PinkShnack]

Here is an example output from nanite/model/core/compute_ancillaries:

• Current version (main)

 Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.44.55.309.jpk-force'[0], 1
no mod ancill: OrderedDict({'max_indent': nan})
no mod ancill: OrderedDict({'max_indent': 6.345093970550771e-07})
no mod ancill: OrderedDict({'max_indent': 6.250291057246192e-07})
no mod ancill: OrderedDict({'max_indent': 6.250291057246192e-07})
no mod ancill: OrderedDict({'max_indent': 6.345093970550771e-07})
no mod ancill: OrderedDict({'max_indent': 6.250291057246192e-07})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.235727936948198e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.235727936948198e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})

 Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.45.26.631.jpk-force'[0], 2
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 6.23558109762272e-07, 'force_jump_max_indent': 1.6543612251060553e-23, 'eta': 9.609520795202152e-14, 'time_ind': 0.11277457672403386})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': 2.661364801610119e-06, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.7578283673631514, 'time_ind': 0.6286059097356625})

• Version with Caching

Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.44.55.309.jpk-force'[0], 1
no mod ancill: OrderedDict({'max_indent': nan})
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 7.940933880509066e-23, 'eta': 4.496924133483972e-13, 'time_ind': 0.11864952963088483})

 Fitting Indentation 'C:\Users\eoconne\Documents\projects\pyjibe\AFM_Data_sample_FOR_EOGHAN\mouse5_H1_30min_ctrl_male\1\force-save-2025.09.19-16.45.26.631.jpk-force'[0], 2
no mod ancill: OrderedDict({'max_indent': nan})
yes mod ancill: OrderedDict({'max_indent': nan, 'force_jump_max_indent': 1.1209325307056298e-09, 'eta': 2.8212658190517192, 'time_ind': 0.6006547477353467})

[paulmueller]

Overall, this sounds like a bug. Fitting should not get slower with number of curves analyzed (A sanity check would be to change the order of curves, but chances are low this is data-related).

It could also be that this is a UI issue (widgets being created and possibly redrawn after every fit).

To answer your questions:

1. Is the rating system deterministic?

Yes. It's based on decision trees.

2. Is it correct that the ancillary parameter calculation get progressively slower as pyjibe runs through the files? Is this due to the curves needing the previous curve calculations because they are one dataset? Or is it just a bug?

This sounds like a bug. Maybe there is a callback or somesuch that triggers computations with previous curves. Did you look into memory usage?

3. The ancillary parameters seems to bounce between very small values and the values they land on, presumably because they are being fit between an interval.

This sounds like mechanics of a specific fitting algorithm. Did you try a different fitting algorithm? It could also be that there are function calls that should not happen (I assume you print-debugged the model function).

Regarding the Caching TODO

Yes, be careful here. Ancillary parameter calculation can involve many parameters. The task is to take into account any possible parameter in the caching key. An alternative strategy would be to do caching in the KVM model (not sure whether that's what you tried), where you have full control over what is relevant.

[PinkShnack]

Hey thanks for the detailed reply. It seems the calc ancillary params is called 4 times per KVM fit:

1. Basic Hertz fit: guess ancill params -> max indent is nan
2. Basic hertz fit: gets max indent
3. KVM fit: i assume it guesses with max indent
4. KVM fit: full fit.

Then subsequent calls do this, but 4 more time and so on. So the 3rd curve runs calc ancill params 12 times, 4th 16 times etc (O(N²)). It seems it is not a recursion issue from nanite (which I assumed).
It is not a CPU or memory issue.

I had tried caching, but I did it on the wrong level (NaniteFitModel). Which meant that when max_indent was calculated as nan (as expected), it kept this value.

Chatgpt pointed out I should do it on the Indentation (curve) level. Which is obvious, but answers always are! ;)

Here is a summary from chatgpt on the root cause, which I implemented in nanite and it speeds up calculation by a lot.

Root cause

• get_ancillary_parameters() was called:

  • during GUI updates
  • during parameter propagation
  • before and after fitting

• compute_ancillaries() depends on post-fit state (params_fitted, contact_point).

• Early (pre-fit) calls returned NaN values (e.g. max_indent), which is expected.

• Because there was no cache invalidation, each new curve caused:

  • repeated ancillary recomputation
  • including for previously processed curves.

• Repeated recomputation masked the fact that ancillaries were being evaluated too early in the fit lifecycle.

Fix to be implemented in nanite:

• move ancillary caching to the Indentation level (per curve).
• introduce a cache (_anc_cache) and validity flag (_anc_valid).
• After a successful fit (fit_model()), the ancillary cache is explicitly invalidated.
• Ancillaries are cached only after a successful fit, based on fit_properties.

NOTE: The results are VERY similar between old and new. I will include some examples in the nanite PR.

[paulmueller]

Curious, I would have thought missing cache invalidation to not be the root problem. I would have expected that the problem was limited cache size. But moving the cache to the Indentation level seems sensible.

Looking forward to AFM-analysis/nanite#27 ❤️