[Major Feature Proposal & Discussion] High Efficiency API Request System

[Googolplexed0]

UPDATE

New efficient-api branch is live, give it a shot by running pipx install git+https://github.com/Googolplexed0/zotify.git@efficient-api.

Goals:

• Significantly reduce / entirely prevent API timeouts (failed to fetch audio key! error)
• Do not require idling / waiting / cooldowns
• Write concise, easily readable and maintainable code

Paths:

• Reduce API calls as much as possible
  • Always use bulk calls if available
  • Carry down sub-item metadata included in parent metadata request
• Use custom class storage objects for API responses
• Alternate between fastest-to-download and slowest-to-download items
  • Time downloading == Time not calling API -> decrease average APIs/sec over total run

Benefits:

• Fewer API calls per object (see Expected # Reduction in API Calls chart)
• Overall faster execution
• Rewritten logic (hopefully) ensures no missed edge cases
• Fewer errors/bugs into the future
• More "pythonic" -> More likely to receive contributions from community members
  • Better community support and upkeep likelihood

Table: Expected Reduction in API Calls for Metadata (shown in ~big O notation)

(n = # objects needed to fetch, t = # contained tracks, a = # contained albums, n, t, a > 0)
(*** = container metadata arrives with 20 fully-populated items)

Object Type	# of Calls (Currently)	# of Calls (Theoretical)	Approximate Reduction
Tracks	n	1 + (n - 1) // 100	100:1
Playlists	n(2 + t)	n[2 + (t - 101)*** // 100]	100:1
Albums	n(2 + t)	[1 + (n - 1) // 20 ][2 + (t - 21)*** // 50 ]	100:1
Artists	2n + 2na + nat	[1 + (n - 1) // 50 ] + n[1 + (a - 1) // 50 ] + na[1 + (t - 1) // 50 ]	75:1

Example Reductions	# of Calls (Currently)	# of Calls (Theoretical)
60 Tracks	60 calls	1 + 0 = 1 call
180 Tracks	180 calls	1 + 1 = 2 calls
1 Playlist, 10 Tracks	1(2 + 10) = 12 calls	1(2 + -1) = 1 call
1 Playlist, 50 Tracks	1(2 + 50) = 52 calls	1(2 + -1) = 1 calls
1 Playlist, 150 Tracks	1(2 + 150) = 152 calls	1(2 + 0) = 2 calls
10 Playlist, 250 Tracks Each	10(2 + 250) = 2,520 calls	10(2 + 5) = 70 calls
3 Albums, 15 Tracks Each	3(2 + 15) = 51 calls	(1 + 0)(2 + -1) = 1 call
30 Albums, 150 Tracks Each	30(2 + 150) = 4,560 calls	(1 + 1)(2 + 2) = 8 calls
2 Artists, 5 Albums Each, 30 Tracks Each	2(2) + 2(2)(5) + (2)(5)(30) = 324 calls	(1 + 0) + 2(1 + 0) + 2(5)(1 + 0) = 13 calls
10 Artists, 35 Albums Each, 20 Tracks Each	2(10) + 2(10)(35) + (10)(35)(20) = 7,720 calls	(1 + 0) + 10(1 + 0) + 10(35)(1 + 0) = 361 calls

Tradeoffs:

• Users cannot choose download order N/A, this will be optional as a new config option.
  • Required for Path #3
• Metadata classes heavily rely on inheritance, which can be hard to read / understand / maintain
  • Base classes may be hard to understand, but the metadata-holding-classes are straightforward
  • Future changes should be metadata-holding-class specific
• Others I may have missed (comment them below)

Discussion:

Is this something you guys believe would be worth coding and integrating? It would require a major rework similar to the original repo's v1.0-dev branch. I don't mind the effort investment, but I don't want to waste my time if no one would find it useful. Are rate limits a frequent pain point for you? Does download order being semi-random bother you? Is there some other tweak that should be considered to further reduce API requests usage? Please discuss with me and each other in the comments below.

[chr15t0ph]

This is a VERY good idea. And would be a MAJOR improvement.

(And: Yes, rate limits are a frequent pain point for me. Your calculations are also related to the time needed for skipping existing files in case of restarting downloads, as described in my other ticket.)

The only downside seems the significant implementation effort for refactoring lots of existing code? "Chapeau" that you not mind the effort for investment.

[DerKO9]

Download order does not matter to me. As long as the exported playlist files have the correct order of songs in them, I'm happy. If not, it is not a big deal, the improved performance would still be welcome.

I'm not a python guy and have been hesitant to help make PRs of features id like, but if the code ends up being more readable, it may make the project more approachable for people like me to help out

[trustosas]

I don't mind rate limits and audio key whatnots. I have 3 different credentials for 3 different accounts for that.
Rate limit = Switch credentials. Audio key errors gone.

[rcMarty]

Since you have a help wanted tag there, maybe you should create detailed subissues with atomized tasks so others can easily help?

[Googolplexed0]

Since you have a help wanted tag there, maybe you should create detailed subissues with atomized tasks so others can easily help?

I finished an alpha version of this proposal and added the help wanted tag so I could request bug testing from users here. Unfortunately, while in the process of getting everything ready to publish, the librespot OAuth/keymaster flow broke (see #48). As a result, I have refrained from releasing the new alpha branch and will wait to do so until everything in main is back in working order.

Anyone who wants to get the new backend published faster can help by working towards an OAuth login5 implementation for librespot-python (again, see #48).

Since #48 has been fixed, the new efficient-api branch will be released soon!

[anbe108]

Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls.

[Googolplexed0]

Persisting terminal session instead of the current behavior of auto exiting zotify after downloading can also be an improvement. ie Currently after user search select & download the program auto exits. This can reduce login calls.

Would be a good add for those using it repeatedly/interactively like an application, but may also detract from those using it as a command line util. Finding a middle ground there may be difficult and I am fairly certain that login calls do not make up a significant number of API requests anyways.