Skip to content

Speed up#3

Open
hjenryin wants to merge 1 commit intoapple:mainfrom
hjenryin:patch-1
Open

Speed up#3
hjenryin wants to merge 1 commit intoapple:mainfrom
hjenryin:patch-1

Conversation

@hjenryin
Copy link

Thanks you for developing this tool!

When I was using it on my data, I found multiple times that the process get stuck when calling smECE(pred,y). In the end, I realized that it was due to the inefficient implementation of np.convolve. I suggest that sp.signal.convolve be used, as it uses FFT to speed up calculation. This difference has also been pointed out in https://stackoverflow.com/questions/53550764/python-numpy-is-there-a-faster-way-to-convolve.

I have attached an example I came across on real data: debug_y_33.5.modify_affix_to_npy.txt
debug_pred_33.5.modify_affix_to_npy.txt (Github doesn't allow .npy files to be uploaded, so please change the affix back to .npy)

The results are:

  • sp.signal.convolve: Time = 0.065s, smECE = 0.006961
  • np.convolve:
    • smECE = 0.006961 as well.
    • It seems that the time spent is not stable. It may vary from one to tens of seconds. But even in the best case, it's still way more time-consuming then the sp.signal implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant