evaluation time of FCS

[TransitionMetal]

Hello.
I'd like to ask how FCS evaluation time depends on number of bins?
I'm trying to process 5-min 400MB file and for
fcs_interval_length = 1
1ns-1us, 1ns-10us, 1ns-1ms intervals it takes about 50 seconds, but for 1ns-100ms much longer (haven't got the end). Any idea?

[snizzleorg]

could you post your code? What do you mean by fcs_interval_length = 1? Are you referring to the interval_length parameter?
https://picoquant.github.io/snAPI/snAPI.Main.html#snAPI_Main_Correlation_setFCSParameters

[TransitionMetal]

could you post your code? What do you mean by fcs_interval_length = 1? Are you referring to the interval_length parameter? https://picoquant.github.io/snAPI/snAPI.Main.html#snAPI_Main_Correlation_setFCSParameters

Exactly, interval_length. 100us-100ms executes in 50 sec, 10us-100ms - in 580 sec. The code is like in example:

from snAPI.Main import *
import matplotlib
matplotlib.use('TkAgg',force=True)
from matplotlib import pyplot as plt
print("Switched to:",matplotlib.get_backend())
import time
import numpy as np
import tkinter as tk
from tkinter import filedialog
import time

# define main func to execute
def main():
    # correlates the data from a file and plots the result
    # call the window of open file dialog
    file_path = filedialog.askopenfilename(
        title="Выберите файл",
        initialdir=r'W:\\',  # \\ to protect '
        filetypes=[('ptu', ['*.ptu', '*.spc'])])  # to see which files are already processed
    # r means raw string with \

    # what measurement to execute
    do_timetrace = False
    do_FCS = True
    do_G2 = False

    # FCS parameters
    fcs_window_size = 1e11
    fcs_interval_length = 1
    fcs_resolution = 1e7

    # create the api object
    sn = snAPI(libType=LibType.TH260)
    # set device to file
    sn.getFileDevice(file_path)

    start_time = time.time()

    # FCS of the data from a file and plotting the result
    
    if do_FCS:
        sn.correlation.setFCSParameters(1, 2, fcs_window_size, fcs_resolution, fcs_interval_length)
        sn.correlation.measure()

        while True:
            finished = sn.correlation.isFinished()
            data, bins = sn.correlation.getFCSData()
            if len(data) > 0:
                plt.clf()
                plt.plot(bins, data[0], linewidth=2.0, label='AB')
                plt.plot(bins, data[1], linewidth=2.0, label='BA')
                plt.xlabel('Time [s]')
                plt.xscale('log')
                plt.ylabel('FCS')
                plt.legend()
                plt.title("FCS")
                plt.pause(1)
                np.savetxt(file_path[:-4] + "-" + str(int(fcs_window_size / 1e9)) + "ms-" + str(
                    fcs_interval_length) + "-FCS.spc",
                           np.column_stack((bins, data[0], data[1])))
            if finished:
                break
        plt.show # (block=True)

    end_time = time.time()
    execution_time = end_time - start_time

    print("Execution time:", execution_time, "seconds")

if __name__ == '__main__':
    main()

[snizzleorg]

I think you are timing the whole plotting operation ... and this is a loop.
so the more iterations the longer it will take.

Could you try this instead:

from snAPI.Main import *
import matplotlib
matplotlib.use('TkAgg',force=True)
from matplotlib import pyplot as plt
print("Switched to:",matplotlib.get_backend())
import time
import numpy as np
import tkinter as tk
from tkinter import filedialog
import time

# define main func to execute
def main():
    # correlates the data from a file and plots the result
    # call the window of open file dialog
    file_path = filedialog.askopenfilename(
        title="Выберите файл",
        initialdir=r'W:\\',  # \\ to protect '
        filetypes=[('ptu', ['*.ptu', '*.spc'])])  # to see which files are already processed
    # r means raw string with \

    # what measurement to execute
    do_timetrace = False
    do_FCS = True
    do_G2 = False

    # FCS parameters
    fcs_window_size = 1e11
    fcs_interval_length = 1
    fcs_resolution = 1e7

    # create the api object
    sn = snAPI(libType=LibType.TH260)
    # set device to file
    sn.getFileDevice(file_path)


    # FCS of the data from a file and plotting the result
    if do_FCS:
        start_time = time.time()
        sn.correlation.setFCSParameters(
            1, 2, fcs_window_size, fcs_resolution, fcs_interval_length
        )
        sn.correlation.measure(waitFinished=True)
        end_time = time.time()
        execution_time = end_time - start_time

        data, bins = sn.correlation.getFCSData()
        if len(data) > 0:
            plt.clf()
            plt.plot(bins, data[0], linewidth=2.0, label="AB")
            plt.plot(bins, data[1], linewidth=2.0, label="BA")
            plt.xlabel("Time [s]")
            plt.xscale("log")
            plt.ylabel("FCS")
            plt.legend()
            plt.title("FCS")
            plt.pause(1)

        plt.show  # (block=True)

    print("Execution time:", execution_time, "seconds")


if __name__ == "__main__":
    main()
    

[tsbischof]

If I am following the thread properly, you are running into the fact that photon correlation scales roughly as O((integration time T)(maximum tau)), so each order of magnitude in maximum tau increases the cost by an order of magnitude. That results in a long execution time for the ms-scale tau you need for FCS

If you can work with approximate results (i.e. just need the shape of the curve) there is an implementation of a multi-tau algorithm in Python here: https://github.com/FCS-analysis/multipletau. I also have a version here in C (photon_intensity_correlate), although with a more low-level interface: https://github.com/tsbischof/photon_correlation

The multi-tau algorithm is designed to cope with the long timescales by grouping photons into bins and changing the effective resolution logarithmically as tau increases. This achieves a speedup from something like O(T*tau) to O(T/dt) for dt as the time resolution, at a cost of losing resolution of individual photon events. Note that this also how hardware FCS correlators tend to work.

There are also some papers noting how to combine the photon correlation at short tau with the intensity/multi-tau correlation at long tau.

If you do need to have photon-level resolution of the signal at long tau, you can also restrict the range of possible tau to reach an acceptable execution time. That is how we built the plots in https://pubs.acs.org/doi/abs/10.1021/nl5027953 and I am glad to share some example scripts if that is the end goal.

[TransitionMetal]

@tsbischof Thank you for your detailed answer! It helps)
And from your experience is there any difference if resolution is decreased at long tau? Because I wouldn't expect any.

[tpoint75]

Of course, we are using a multi tau algorithmus to calculate the FCS. Just look at the bins array or the documentation:
https://picoquant.github.io/snAPI/snAPI.Main.html#equation-multitau
But anyway if you have a problem of the calculation time just reduce the amount of data to calculate. If you want to se the FCS between 10us-100ms why you need over 300s of data? 100ms fits 3000 times in this area!

With this loop:
if do_FCS:
start_time = time.time()
sn.correlation.setFCSParameters(
1, 2, fcs_window_size, fcs_resolution, fcs_interval_length
)
sn.correlation.measure(waitFinished=True)
end_time = time.time()
execution_time = end_time - start_time

    data, bins = sn.correlation.getFCSData()
    if len(data) > 0:
        plt.clf()
        plt.plot(bins, data[0], linewidth=2.0, label="AB")
        plt.plot(bins, data[1], linewidth=2.0, label="BA")
        plt.xlabel("Time [s]")
        plt.xscale("log")
        plt.ylabel("FCS")
        plt.legend()
        plt.title("FCS")
        plt.pause(1)

    plt.show  # (block=True)

..you can see how the FCS does change over the calculation time. Also you can add a sn.correlation.clearMessage() to delete the calculated data. than you see how the FCS changes over the data set. It does not make sense to average these Correlation over such a long time if the signal changes that much.

But anyway, I a faster way to calculate the FCS is with Symphotime64. The have a much better algorithm that is using the gpu. I will look what we can do, to make it more performant in the future..

[tpoint75]

@TransitionMetal I could improve the FCS algorithm by using a better multi tau implementation that is significant faster longer window times and higher count rates.
It will be implemented in the next update!

[TransitionMetal]

@TransitionMetal I could improve the FCS algorithm by using a better multi tau implementation that is significant faster longer window times and higher count rates. It will be implemented in the next update!

that's good news, thank you! Looking forward to release)

[tpoint75]

Hi @TransitionMetal I released the new FFCS method witch should speed up this significantly. Please report if its fixed your issue!

[tpoint75]

https://picoquant.github.io/snAPI/snAPI.Main.html#snAPI_Main_Correlation_setFFCSParameters

[TransitionMetal]

@tpoint75 Thank you, I found it two times faster on my file!

[tpoint75]

@TransitionMetal have you tried to increase fcs_interval_length to 8, which should be the minimum for FCS?

[TransitionMetal]

@tpoint75 Yes, I used fcs_interval_length = 8