[WIP]: V3

MIGRATE.md

@@ -0,0 +1,47 @@
+# Migration from v2 to v3
+
+## Overview
+
+I had to refactor the website to accomodate OnSIDES v3.
+The database schema changed and we added data from different sources.
+Also, we wanted to let users filter by product, ingredient, and adverse effect.
+To make this happen, I was transitioning the project from Flask+NextJS page router to just NextJS app router, which should simplify the app a lot.
+I made a bunch of pages already that have the additional functionality (e.g. filter by source).
+Unfortunately, I didn't finish.
+If you have any questions, I'll be online ([@zietzm](https://github.com/zietzm)).
+Feel free to Slack me or tag me in an issue.
+Good luck.
+
+## History
+
+- V1 of the website used Flask for a JSON API, a statically-exported `create-react-app` frontend, and a MySQL database ([dhvanim](https://github.com/dhvanim) in 2022).
+- V2 of the website used a faster, simplified Flask backend, a statically-exported `create-next-app` frontend, and a SQLite database ([zietzm](https://github.com/zietzm) in 2024).
+- V3 is in progress. My (@zietzm) goal was to move to a full-stack NextJS app (from pages router to app router), but I wasn't able to finish this before leaving (May 2025).
+
+## Remaining Todos
+
+- Build a derived table to serve the individual ingredient page.
+- Update the individual ingredient page component.
+- Fix the frontpage stats query (this was another derived table that needs to be built).
+- (If you stick with full-stack JS) remove the Flask backend, greatly simplify the deployment playbook, consider using [Caddy](https://caddyserver.com/) instead of NGINX for even greater simplicity (auto SSL certs, much simpler config).
+
+## Why full-stack JS?
+
+Compared to v2, OnSIDES v3 has WAY more data and uses a new database schema.
+Previously, the backend would give full lists (e.g. all ingredients, all adverse effects, etc.).
+This stopped being feasible with v3 data, so I started trying to implement paging.
+But, the new database format made the queries too slow, and I needed deterministic ordering.
+So I started working on SQL queries to build derived tables to speed up queries (see [frontend/derived-tables.sql](frontend/derived-tables.sql)).
+This seems to have worked well for the pages that have already been updated.
+Unfortunately, I wasn't able to finish this work for the most complex pages, like the individual ingredient pages.
+
+## Apologies
+
+I've only ever built web apps with JSON APIs and React frontends or full HTML templates with SSR.
+This whole SSR React thing seemed like a good choice for this project, and may still be, but it certainly complicated this refactor.
+I was using [Bun](https://bun.sh/) for this project, and inadvertently made it dependent on Bun by using the [Bun SQLite driver](https://bun.sh/docs/api/sqlite).
+This should be an easy fix if you want to swap it.
+
+If you're refactoring this project, I'm not sure I'd suggest sticking with the Bun+NextJS combo.
+A lot of other lab projects use Remix, which seems nice.
+The hardest part of refactoring this should be developing the last SQL queries.

README.md

@@ -28,7 +28,10 @@ tar -xzvf onsides_v2.1.0_20240925.tar.gz
 ```
 
 To create the database, use the `etl/build-database.sh` file.
-You'll need to modify the first line to reference the path to the unpacked data directory.
+**You'll need to modify the first line to reference the path to the unpacked data directory.**
+```bash 
+bash etl/build-database.sh
+```
 This will take a few minutes to run.
 Once complete, the `database.db` file should be ~300 MB.
 

etl/build-database.sh

@@ -6,5 +6,6 @@ rm -f loaded.db
 
 cat ../../etl/load.sql | duckdb
 cat ../../etl/format.sql | sqlite3 loaded.db
+cat ../../etl/derived-tables.sql | sqlite3 loaded.db
 
 mv loaded.db ../../database.db