[389] Add support for bio data ingest from postgres db

[dbashford-NOAA]

Resolves #389

• Added a function to load bio data from a postgres db as an alternative to loading from a CSV
• Uses psycopg package to connect to postgres

[leewujung]

Quick recap from email exchanges between me and @dbashford-NOAA :

• the current workflow has been tested (separately from the rest of the workflow) to pull data from the existing db in AWS set up by @aliciabillings-noaa
• @dbashford-NOAA will create a mock db to simulate the connection to enable having tests for this new functionality

[brandynlucca]

@dbashford-NOAA I see that you are running into similar issues with jupyter-book as I did when drafting #416 earlier this week. Jupyter Book put out a major release this week that made rendering the RTD pages a frustrating experience. The easiest fix is to pin jupyter-book==1.0.4.post1, which will allow the docs test to pass and doesn't require a complete overhaul of the configuration files. I believe you will also have to pin docutils>=0.18,<0.22 [ref], but that may have been the source of an unrelated error on my end.

[dbashford-NOAA]

@brandynlucca Thanks for the help! I ended up pinning jupyter-book and docutils to resolve it. Definitely frustrating to troubleshoot.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[dbashford-NOAA]

@leewujung @brandynlucca It looks like the tests are failing because of the psycopg dependency for my changes.
I've added it to requirements.txt but I only see requirements-dev.txt in the PR workflow.
Do I need to add psycopg somewhere else for the workflow?

[brandynlucca]

@leewujung @brandynlucca It looks like the tests are failing because of the psycopg dependency for my changes. I've added it to requirements.txt but I only see requirements-dev.txt in the PR workflow. Do I need to add psycopg somewhere else for the workflow?

You need to also add it to pyproject.toml.

[dbashford-NOAA]

@leewujung @brandynlucca
These changes pass testing fine in my local dev environment but I'm having some trouble with passing for each platform.
Testing with a local postgres db has been the main part of the issue.
My first testing implementation was using testing.postgresql to setup a temporary db with test data which works fine in my local environment and passes testing on macos and linux but not windows because of permissions issues.

Execution of PostgreSQL by a user with administrative permissions is not permitted.
The server must be started under an unprivileged user ID to prevent possible system security compromises. 
See the documentation for more information on how to properly start the server.

There may be a way around this by running the windows tests in the workflow as a non-admin user but that could cause problems with other part of the process.

I also tried setting up a docker container service to run the postgres db in the workflow but that's only supported for linux so the tests failed for macos and windows.

Container operations are only supported on Linux runners

Another solution could be to have an external postgres db setup to test against but I'm not sure where that would live or how we would manage credentials.

Let me know if you have any ideas!

[leewujung]

@dbashford-NOAA : this is interesting - looks like this action would be helpful? https://github.com/marketplace/actions/setup-postgresql-for-linux-macos-windows

[dbashford-NOAA]

@leewujung
It took some testing to figure out but I got that action to work and fixed the issue.
It's passing all the tests and is ready for review now.

[leewujung]

@dbashford-NOAA : Awesome! I am glad it worked out! Looking forward to reviewing this (and all the other reviews I owe you...).

[leewujung]

Hey @dbashford-NOAA: Thanks for the PR and making postgres tests in CI work!

I think this PR is almost ready to go, just have some consistency issues that I marked in inline comments.

In addition, how about changing the biodata loading function names so that it is clear where the data are from?

• load_biological_data --> load_biological_data_excel
• load_db_biological_data --> load_biological_data_database

[dbashford-NOAA]

@leewujung
I've made those changes so it should be ready now!

[leewujung]

@dbashford-NOAA : Great, thanks! Everything looks fine. I will merge this after the hackday tomorrow in case of breaking something due to the function name change for the bio data ingestion part.

[leewujung]

For visibility: After the hack day we found out that the database that has been worked on was not the correct one, so more changes is needed on the database end. I'll hold on to reviewing / merging this until that is resolved. We have a meeting tomorrow (Jan 30) that will hopefully resolve this.

[ElizabethMPhillips]

Following up on this and after checking with Wu-Jung today, @dbashford-NOAA please ensure that the updated materialized views in the nwfsc_public_prod.acoustics schema match the previous views developed, and then go ahead and update the
code to access the nwfsc_public_prod.acoustics schema. After that is complete, we can test pulling the new, merged length view (that includes both length and specimen in one table)

[aliciabillings-noaa]

Let's flesh this out at our meeting tomorrow. I thought we agreed to use trimmed down versions of just catch and specimen (that includes all length fish as well).

…

-- Alicia Billings, M.S. Fishery Biologist NOAA/NMFS/NWFSC/FRAM/FEAT 2032 Marine Science Drive Newport, OR 97365 541.275.1742 Google voice

On Mon, Feb 2, 2026 at 2:06 PM Elizabeth Phillips ***@***.***> wrote: *ElizabethMPhillips* left a comment (OSOceanAcoustics/echopop#415) <#415 (comment)> Following up on this and after checking with Wu-Jung today, @dbashford-NOAA <https://github.com/dbashford-NOAA> please ensure that the updated materialized views in the nwfsc_public_prod.acoustics schema match the previous views developed, and then go ahead and update the code to access the nwfsc_public_prod.acoustics schema. After that is complete, we can test pulling the new, merged length view (that includes both length and specimen in one table) — Reply to this email directly, view it on GitHub <#415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGZ752KVC2MXI7C2FSPNLVD4J7C67AVCNFSM6AAAAACKZZIAZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQMZXGU4DKNJSGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[leewujung]

Let's flesh this out at our meeting tomorrow. I thought we agreed to use
trimmed down versions of just catch and specimen (that includes all length
fish as well).

Yes - this is confirmed before/at the Feb 19 meeting.

From Feb 19 notes, these below can be added to the materialized view:

• Will add SPECIES_CODE into the view
• Can try to generate UID in the materialized view

Current UID: {SHIP}-{SURVEY_ID}-{SPECIES_CODE}-{HAUL_NUMBER}

Also @dbashford-NOAA : How are the views from the database coming along? Do you think you can finish this soon? There are recent codebase changes, so please merge the upstream to your PR. Since it's been so long and there are going to be new changes coming up from @brandynlucca to use the catch and specimen, to avoid confusion, I think it'll be better if you revert the function name load_biological_data_excel back to load_biological_data. Thanks.

[dbashford-NOAA]

@leewujung
I've updated the ingest and testing code to not expect the length view but was waiting on the changes from @brandynlucca before finalizing it to make sure it'll be compatible. I don't think any further changes will take much time though.

I'll work on pulling in the upstream changes and I can revert the function name too.

[leewujung]

Sounds good - thanks @dbashford-NOAA !

[leewujung]

@dbashford-NOAA : Just a quick note that once #463 is merged you could go ahead to make the changes. That PR should include all the handling needed to use the single specimen file (combining the previous specimen and length files) plus the catch file for running the workflow.