Cache only selected result in a search

[jghibiki]

I've noticed that when we make a search, sonicflow appears to be caching every item returned by grooveshark. I think that only caching the result selected for play by the user would make more effective use of the cache storage, but in addition would result in many less-relevant items from being cached to sonicflow (and hopefully making better use of the updated search algorithm more below on this). This idea follows two assumptions,

1. Songs that have been played before should be considered "more relevant" as search results than those not explicitly played. Also, a song that has been played at least once, has a higher chance of being played again than never being played again.
2. Other search results returned by grooveshark may be relevant but not notable or confirm-ably relevant to be worth caching on the local server.

Impact in combination with the search algorithm:

I have noticed that the new search algorithm produces notably more relevant results, but almost only when going to grooveshark for results. When searching the local cache, the results seem relevant only down to a single word or two with the truly relevant results hidden in a long list of fakes.

[jghibiki]

I guess another option would be to do better sorting of the cached results with javascript so the work would be done client side.

[dekarrin]

This is not acutally a problem with the search algorithm, though commit 40d6801 seems to have partially solved the relevancy issue.

[weikanglim]

It seems like multiple issues were raised by @jghibiki, while the relevancy issue has been resolved, there hasn't been any changes to the caching algorithm. I agree completely with the suggestions from @jghibiki. Caching only requested songs will decrease the search time significantly for the user since currently the user is also waiting for all searched songs to be added to the database. I agree that the following features should be implemented:

• Cache only requested songs.
• Store play counts of each song in SonicFlow.
• Revamp local search to take into account play counts.

[zeande]

I think it's a good idea to store play counts and incorporate that into the search algorithm. However, although it might be worth trying, I don't think it will pay off to only store the selected song in the cache. The primary purpose of even having the metadata cache in the first place was to reduce calls made with the Grooveshark/TinySong APIs, which were very expensive. By caching all of the results, we can greatly reduce the number of these API calls.

If the end user has to wait for the songs to be added to the cache (which I really don't think should be taking that much additional time anyway, since we already have the information in memory), then I think a better solution would be to display the results and then add the songs to the cache. Or do it asynchronously.

[weikanglim]

I wasn't sure how much the song insertions was effecting the search times, because we could be potentially searching for existing records (although by primary keys), and inserting 100+ rows at a time.

The issue that I have been having is search times on order of seconds, while searching a song on Grooveshark takes less than a second. Granted, groovesharkpy seems to be reverse engineering the API which could potentially take a longer time, but I'm still not sold on it taking more than a second. I'll research the issue further and see if I can find the cause of these slow search times.

Also, what do you think about the suggestion: only caching search results for search by album / artists?

EDIT: Perhaps the issue is the number of queries that are involved in song caching. Perhaps making a couple of batch queries might be more efficient, instead of the potentially 100+ queries:

1. Build lists of songs, artists, albums from search results.
2. Search for all potentially existing songs, artists and albums.
3. Batch insert all non-existing songs.

[zeande]

Searching a song on Grooveshark takes less than a second

Are you sure about that? When I search for a song (that exists in Sonicflow), it takes less than a second. When I click the button that says "Song not listed?" to search Grooveshark, it takes several seconds.

Try it with the song title: "Dalai lama". Two results instantly show up from our cache. But when you click "Song not listed?" it takes a few seconds to bring back Grooveshark results. You can see similar results even if you further narrow it down by artist (Rammstein).

[weikanglim]

To clarify, I meant searching on grooveshark.com, not through our system, and I'm thinking that it might be a problem on our end that searches from Grooveshark are slow.

[dekarrin]

We need to be sure that it's specifically the Grooveshark search that is slowing down SonicFlow. Granted, groovepydown is reverse engineering the search, but as far as I can tell it's making the same calls that your browser does, so it shouldn't affect the search speed.

I know that originally we were making many calls to third-party APIs for song/album/art information. This never really panned out (not perfectly anyways), and I suspect that these calls (to TinySong/Gracenote/whatever) are what is taking the majority of the time.

Say you return 200 results from a grooveshark search (The maximum grooveshark will return). If I'm not mistaken, this means 200 individual calls to third party databases for song metadata. If that is the case, it is likely that this is what is slowing down a grooveshark search, and not the grooveshark call itself. I believe that the code should be analyzed to see if this is happening.

[zeande]

Ah, I see what you are saying now. Yeah, it is very fast on their website, and analyzing how we're using the APIs is a good idea.

[dekarrin]

Do you recall whether the third-party APIs are actually useful? I don't think that we're actually using that data. If not, we should cut out the calls completely.

[zeande]

I've been looking through the source code for a while now, and I don't see any calls to GraceNote for example. Are you talking about any calls in specific made to third-party APIs?

Also, what API does pygs use? I see it references "grooveshark", but where is that library?

[dekarrin]

pygrooveshark is an external python library: https://github.com/koehlma/pygrooveshark

Originally my thinking was that we shouldn't include the source for that, since the user can simply grab it from koehlma's GitHub at the latest version.

And as far as API calls go, I'm not sure which ones are made. Perhaps a better analysis can be done by using the timing functions in PHP to analyze execution time and use a manual binary search to identify the code that is taking the most time? That's how I originally cut down the time for a SonicFlow cache search.

[weikanglim]

@dekarrin I'll be willing do that. Currently finding a suitable machine to start building the project.