[Proposal] Migrate the dev/CI Jupyter stack to Notebook 7

[jlarson4]

Proposal

Migrate the jupyter dependency group to Notebook 7 (built on Jupyter Server 2 / JupyterLab 4). This proposal is a follow-up to #1437.

Motivation

PR #1437 fixed the ModuleNotFoundError: No module named 'jupyter_server.contents' (jupyter/notebook#7048) by bumping the locked notebook from 6.5.4 to 6.5.7. That was the correct fix, but Notebook 6.5.5+ pins jupyter_client<8. The bump pulled the kernel stack backwards, jupyter_client 8.6.3 was rolled back to 7.4.9 and ipykernel 6.30.0 was rolled back to 6.29.5. We are now on a maintenance-only/EOL notebook line carrying an old jupyter_client.

Pitch

#1437 traded a modern jupyter_client/ipykernel for a working jupyter notebook launch. Notebook 7 lets us have both.

Why Notebook 7

• Maintains the fix from Fix error opening jupyter notebook #1437 – The jupyter_server.contents error came from the legacy classic server's coupling to old jupyter_server internals; Notebook 7 is rebuilt on Jupyter Server 2 and does not hit it.
• Moves us to the modern kernel stack – Notebook 7 works with jupyter_client 8.x, so we get back to 8.6.3 (and current ipykernel). Notebook 6 is maintenance-only/EOL; Notebook 7 is actively maintained.
• Fallback option – Anyone who wants the classic UI can install nbclassic,
  which runs as a Jupyter Server extension and coexists with Notebook 7.

Additional context

After migration, confirm the canonical demos (Main_Demo, Exploratory_Analysis_Demo at minimum) on Colab's current runtime to ensure widget/output rendering is intact after our changes.

Risks / open questions

• Any local dev workflow relying on the classic Notebook UI or classic-only nbextensions?
  If so, document the nbclassic install path.
• Confirm jupyterlab>=4 does not regress circuitsvis / ipywidgets>=8.1.1 rendering in
  the demos.

Acceptance criteria

• jupyter group resolves to notebook>=7 with jupyter_client back at ≥8 and current ipykernel.
• uv run jupyter notebook demos/<x>.ipynb opens without ModuleNotFoundError (Fix error opening jupyter notebook #1437's bug).
• make notebook-test passes.
• make test (full tiers) passes.
• Canonical demos verified on Google Colab.
• No new jupyter_client / pyzmq / notebook pins in demo bootstrap cells.

References

• TransformerLens #1437 — Fix error opening jupyter notebook (notebook 6.5.4→6.5.7)

• jupyter/notebook#7048 — ModuleNotFoundError: No module named 'jupyter_server.contents'

• jupyter/notebook#6749 — Pin jupyter_client<8 and pyzmq<25

• Migrating to Notebook 7

• nbclassic

• I have checked that there is no similar issue in the repo (required)