feat: upgrade to psycopg 3

[dbirman]

I believe this is the correct way to get us out of the mess we're in with Redshift dependencies. I tested with python 3.13 and this works properly for me, and it installs in another repository (zombie-squirrel) that communicates with redshift, and that runs properly.

We should be careful to test this across python versions on non-mac architectures before merging.

[dbirman]

Testing on Python 3.12 on a different mac (M1 instead of M3) this failed for me with 2025-11-19 16:10:04,411 Error fetching from cache: (psycopg.NotSupportedError) codec not available in Python: 'UNICODE' [SQL: select pg_catalog.version()]

[dbirman]

On further testing it just seems like psycopg2/SQLAlchemy don't work well on Mac, and also maybe aren't the proper solution anymore. Trying with redshift-connector instead (and changing to a draft)

[jtyoung84]

For an early 2026 project, I want to run this in a Lambda function and have users interact with an REST API endpoint like what we're doing with DocDB instead of interfacing with Redshift directly. If this is working in Code Ocean at least, then maybe that can hold us over until then.

[dbirman]

redshift-connector seems to work for reading from redshift. I need to test across python versions though and test writing. But yes a different solution is probably better. I will also ask David about just read/write from some parquet files in a folder on aind-open-data as a workaround until a better API can be built.

[jtyoung84]

redshift-connector seems to work for reading from redshift. I need to test across python versions though and test writing. But yes a different solution is probably better. I will also ask David about just read/write from some parquet files in a folder on aind-open-data as a workaround until a better API can be built.

aind-scratch-data is also a public bucket that might be better as a short-term solution.