Feature: API to download scripts

[jthemphill]

Description
Hello! I collaborate on https://botc-tools.xyz. Our website download all of the scripts from botc-scripts and present them in a speedy, mobile-friendly format. We're able to do this because all of the scripts put together are only 1.4MB of JSON! But we can only build this JSON by scraping your site, page by page, which is inefficient for both server and client.

Do you think we could make an endpoint to download scripts?

[AdmiralGT]

I'm confused by this request. There is already an endpoint to download scripts... https://botcscripts.com/api/scripts/

But you're already using that to get the scripts https://github.com/tchajed/botc-tools/blob/main/fetch_assets/src/all_scripts.ts#L20

You're saying it's inefficient, but that's just the database query. I've put together a speculative change to increase the page size you can query at once #501 but even with some local testing a 100 page size was taking a couple of seconds to complete so I'm cautious to just increase it.

I'm also going to question why you're downloading the entire database each time. Only the score field is dynamic in the API response and you don't seem to be using that so it would really be better for you to store a last entry variable each time you do a database download and then only download until you have that entry again.

[jthemphill]

Thank you for your quick response! You've cut to the heart of my request, which is that I really just want to be fetching scripts in the most efficient manner for both of us.

I'm also going to question why you're downloading the entire database each time. Only the score field is dynamic in the API response and you don't seem to be using that so it would really be better for you to store a last entry variable each time you do a database download and then only download until you have that entry again.

Ah, it looks like if a script updates, then it will be bumped to the head of the list? That will work!

[AdmiralGT]

Yeah, "scripts" is a bit of a misnomer here and the default behaviour probably doesn't help with that.

The API by default provides unique Name + Version objects (which in the code to I refer to as ScriptVersion) but it only returns the latest Version (you can request all versions with the parameter all_scripts=on) by default. So when a script is updated, it's moved to the top of the list. I think in reality very few people want to use the API to get all script versions and just get the latest.

This does mean that it's possible that the last entry you got no longer appears in the default response (because the last entry you got was a script that has since been updated) but that should be workable by <=.