Upgrade Python version to 3.12 and update dependencies; replace pandas-datareader with pyfredapi in notebooks.

Author: jtkiley

.devcontainer/devcontainer.json

@@ -1,6 +1,6 @@
 {
     "name": "carmapy",
-    "image": "mcr.microsoft.com/devcontainers/python:1-3.11-bookworm",
+    "image": "mcr.microsoft.com/devcontainers/python:3.12-bookworm",
     "features": {
         "ghcr.io/rocker-org/devcontainer-features/quarto-cli:latest": {
             "version": "prerelease"
@@ -18,6 +18,7 @@
     "postCreateCommand": "zsh ./.devcontainer/post_create.sh",
     "containerEnv": {
         "API_NYT": "${localEnv:API_NYT}",
+        "FRED_API_KEY": "${localEnv:FRED_API_KEY}",
         "WRDS_USER": "${localEnv:WRDS_USER}",
         "WRDS_PASS": "${localEnv:WRDS_PASS}"
     }

.github/workflows/test_notebooks.yml

@@ -4,6 +4,7 @@ on: [push]
 
 env:
   API_NYT: ${{ secrets.API_NYT }}
+  FRED_API_KEY: ${{ secrets.FRED_API_KEY }}
   WRDS_USER: ${{ secrets.WRDS_USER }}
   WRDS_PASS: ${{ secrets.WRDS_PASS }}
 
@@ -13,7 +14,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.11"]
+        python-version: ["3.12"]
 
     steps:
       - uses: actions/checkout@v3

.pre-commit-config.yaml

@@ -5,12 +5,11 @@ repos:
     -   id: black
     -   id: black-jupyter
 -   repo: https://github.com/nbQA-dev/nbQA
-    rev: 1.6.3
+    rev: 1.9.1
     hooks:
     -   id: nbqa-ruff
-        additional_dependencies: [ruff==0.0.256]
--   repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: v0.0.256
+-   repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.9.10
     hooks:
     -   id: ruff
         args: [--exit-non-zero-on-fix]

notebooks/2c_retrieval2.ipynb

@@ -11,7 +11,8 @@
     "In this notebook, we will work with the following:\n",
     "\n",
     "- API access.\n",
-    "- pandas-datareader\n",
+    "    - NYTimes API.\n",
+    "    - FRED API.\n",
     "- Commercial databases.\n",
     "- SQL."
    ]
@@ -27,7 +28,7 @@
     "\n",
     "import plotly.express as px\n",
     "import pandas as pd\n",
-    "import pandas_datareader as pdr\n",
+    "import pyfredapi as pf\n",
     "from pynytimes import NYTAPI"
    ]
   },
@@ -130,12 +131,17 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# pandas-datareader\n",
+    "# FRED\n",
     "\n",
-    "The `pandas-datareader` package provides access to a number of APIs that have a huge amount of data.\n",
-    "It is under very active development, so you may see new sources appear.\n",
+    "Another API example is FRED, the Federal Reserve Economic Data system.\n",
+    "It's a fairly typical API in that you register for a key, and then you can access the data.\n",
     "\n",
-    "Below, we have a simple example of getting the 10-year US Treasury bond yield at monthly resolution from FRED, the Federal Reserve Economic Data system."
+    "Let's try a less-guided example of obtaining your API key.\n",
+    "\n",
+    "- [FRED API key](https://fred.stlouisfed.org/docs/api/api_key.html)\n",
+    "- [pyfredapi documentation](https://pyfredapi.readthedocs.io/en/latest/)\n",
+    "\n",
+    "Below, we have a simple example of getting the 10-year US Treasury bond yield at monthly resolution from FRED."
    ]
   },
   {
@@ -144,10 +150,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "start = datetime.datetime(2012, 1, 1)\n",
-    "end = datetime.datetime(2023, 12, 31)\n",
-    "\n",
-    "fred_gs10 = pdr.DataReader(\"GS10\", \"fred\", start, end).reset_index()\n",
+    "fred_gs10 = pf.get_series(\"GS10\")\n",
     "fred_gs10.head()"
    ]
   },
@@ -157,7 +160,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "px.line(fred_gs10, x=\"DATE\", y=\"GS10\", template=\"plotly_dark\").show()"
+    "px.line(\n",
+    "    fred_gs10[fred_gs10[\"date\"] >= pd.to_datetime(\"2012-01-01\")],\n",
+    "    x=\"date\",\n",
+    "    y=\"value\",\n",
+    "    template=\"plotly_dark\",\n",
+    ").show()"
    ]
   },
   {
@@ -166,7 +174,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "len(fred_gs10)"
+    "len(fred_gs10[fred_gs10[\"date\"] >= pd.to_datetime(\"2012-01-01\")])"
    ]
   },
   {
@@ -207,8 +215,7 @@
     "\n",
     "If time permits, choose one of the following as a group.\n",
     "\n",
-    "1. Experiment with other searches with the NY Times API by adapting the code above. You may want to create new cells below.\n",
-    "1. Look at the [pandas-datareader documentation](https://pandas-datareader.readthedocs.io/en/latest/), and decide on another dataset to try out. Like the option above, you may want to create new cells."
+    "1. Experiment with other searches with the NY Times API by adapting the code above. You may want to create new cells below."
    ]
   }
  ],
@@ -228,7 +235,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.4"
+   "version": "3.12.2"
   },
   "vscode": {
    "interpreter": {

requirements.txt

@@ -1,18 +1,18 @@
-beautifulsoup4==4.12.3
+beautifulsoup4==4.13.3
 geopandas==1.0.1
 geopy==2.4.1
 hvplot==0.11.2
 ipykernel==6.29.5
-jupysql==0.10.16
-nbconvert==7.16.5
+jupysql==0.11.0
+nbconvert==7.16.6
 nbformat==5.10.4
 notebook==7.3.2
 pandas==2.2.3
-pandas-datareader==0.10.0
-plotly==5.24.1
-polars==1.18.0
-pre_commit==4.0.1
-pyarrow==18.1.0
+plotly==6.0.0
+polars==1.24.0
+pre_commit==4.1.0
+pyarrow==19.0.1
+pyfredapi==0.9.1
 pynytimes==0.10.0
-textblob==0.18.0.post0
-wrds==3.2.0
+textblob==0.19.0
+wrds==3.3.0