feat: Added comprehensive unit testing and github action to run tests on new pull requests #8

dklawren · 2026-01-21T22:27:43Z

No description provided.

… on new pull requests

Copilot

Pull request overview

Adds a comprehensive pytest-based test suite and GitHub Actions workflow to automatically run linting, unit tests, and integration tests for the GitHub ETL pipeline.

Changes:

Added a large pytest test suite covering main.py functions (extraction, transformation, loading, orchestration).
Added pytest/coverage configuration and a testing guide documenting local + CI workflows.
Added a GitHub Actions workflow to run linting/tests on pull requests; expanded dependencies to include lint/format tools.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`test_main.py`	New comprehensive unit/integration test suite for `main.py`.
`requirements.txt`	Adds dev tooling dependencies (black/flake8/mypy/isort) alongside existing test deps.
`pytest.ini`	Configures pytest discovery, verbosity, and coverage reporting.
`TESTING.md`	Documents how to run tests/linting locally and in CI, plus docker-based integration testing.
`.github/workflows/tests.yml`	Adds CI workflow for linting, pytest runs (unit + all), coverage artifacts, and docker-compose integration job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test_main.py

pytest.ini

TESTING.md

test_main.py

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

main.py

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

main.py:536

GITHUB_REPOS is split on commas but entries aren’t stripped/validated. Values like "owner/repo, owner/repo" (note the space) or a trailing comma will produce repo strings with leading whitespace or empty entries, which will break API URLs. Consider stripping whitespace and filtering out empty repo names before iterating.

    github_repos = []
    github_repos_str = os.getenv("GITHUB_REPOS")
    if github_repos_str:
        github_repos = github_repos_str.split(",")
    else:
        raise SystemExit(
            "Environment variable GITHUB_REPOS is required (format: 'owner/repo,owner/repo')"
        )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

TESTING.md

.github/workflows/tests.yml

pytest.ini

TESTING.md

cgsheeh

I haven't looked over the tests in full yet, but wanted to leave comments about the higher-level items I noticed off the bat.

.github/workflows/tests.yml

test_main.py

cgsheeh

There is a lot of content in this PR, it might take a few iterations to review it fully.

We could land some of this code faster if we split it into smaller chunks. For example, we could add linting support and tests in one PR, and deal with the tests themselves in another.

cgsheeh · 2026-01-26T16:51:42Z

pyproject.toml

+requires-python = ">=3.14"
+license = {text = "MPL-2.0"}
+authors = [
+    {name = "Mozilla", email = "dev-platform@lists.mozilla.org"}


Suggested change

{name = "Mozilla", email = "dev-platform@lists.mozilla.org"}

{name = "Mozilla", email = "conduit-team@mozilla.org"}

cgsheeh · 2026-01-27T23:30:57Z

test_formatting.py

+
+
+def test_ruff():
+    passed = subprocess.call(("ruff", "check", "main.py", "--target-version", "py314"))


You can remove the --target-version and it will be inferred from the requires-python in pyproject.toml.

Also, you can specify the root of the repo to ensure we lint all files going forward.

Suggested change

passed = subprocess.call(("ruff", "check", "main.py", "--target-version", "py314"))

passed = subprocess.call(("ruff", "check", ".",))

cgsheeh · 2026-01-27T23:31:19Z

test_formatting.py

+
+
+def test_black():
+    cmd = ("black", "--diff", "main.py")


Suggested change

cmd = ("black", "--diff", "main.py")

cmd = ("black", "--diff", ".")

cgsheeh · 2026-01-28T16:09:57Z

test_main.py

+
+import main
+
+# =============================================================================


We should create a tests/ directory for all the test files, and in that directory we should have a conftest.py file where we define all the fixtures. This is the common configuration for testing Python with pytest.

cgsheeh · 2026-01-29T17:44:47Z

test_main.py

+# =============================================================================
+# TESTS FOR EXTRACT_COMMITS
+# =============================================================================
+


Is this supposed to be a new test? Missing def test_name(): here if so.

Can you remove these long # ===== page separators? If you feel they are necessary, you should instead split these tests out into their own file like test_extract_commits.py.

cgsheeh · 2026-01-29T17:55:13Z

TESTING.md

@@ -0,0 +1,621 @@
+# Testing Guide for GitHub ETL


This file reads like an AI generated description of the PR. Let's remove it, or update it to only include instructions on how to run various testing commands. A doc that says "here are the tools we use and our coverage numbers" isn't very useful and is likely to bitrot quickly, as you can see from the fact that it still references isort, mypy and others despite already switching to ruff.

feat: Added comprehensive unit testing and github action to run tests…

da8cb65

… on new pull requests

dklawren requested a review from Copilot January 21, 2026 22:27

Copilot started reviewing on behalf of dklawren January 21, 2026 22:28 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

dklawren added 7 commits January 21, 2026 18:47

Copilot suggested fixes

89d1edb

Fix integretion test

43b13f0

Black formatted

6042681

Used isort to fix sorting order

3943582

Mypy test fixes

483f19b

types-requests

af0db81

More types-requests fixes

9b3ba75

dklawren requested a review from Copilot January 22, 2026 02:55

Copilot started reviewing on behalf of dklawren January 22, 2026 02:55 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

main.py Outdated Show resolved Hide resolved

Fixed typo in f-string

eaf389b

dklawren requested review from cgsheeh and Copilot January 22, 2026 03:12

Copilot started reviewing on behalf of dklawren January 22, 2026 03:12 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

TESTING.md Outdated Show resolved Hide resolved

TESTING.md Outdated Show resolved Hide resolved

TESTING.md Show resolved Hide resolved

.github/workflows/tests.yml Outdated Show resolved Hide resolved

pytest.ini Outdated Show resolved Hide resolved

TESTING.md Outdated Show resolved Hide resolved

Copilot fixes

7caae20

cgsheeh requested changes Jan 23, 2026

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

.github/workflows/tests.yml Outdated Show resolved Hide resolved

test_main.py Outdated Show resolved Hide resolved

dklawren added 2 commits January 23, 2026 18:16

Fixed review comments

3b9260d

Fixed action to install testing dependencies

cb582d3

dklawren requested a review from cgsheeh January 23, 2026 23:22

cgsheeh requested changes Jan 29, 2026

View reviewed changes

	{name = "Mozilla", email = "dev-platform@lists.mozilla.org"}
	{name = "Mozilla", email = "conduit-team@mozilla.org"}



		def test_ruff():
		passed = subprocess.call(("ruff", "check", "main.py", "--target-version", "py314"))

	cmd = ("black", "--diff", "main.py")
	cmd = ("black", "--diff", ".")


		import main

		# =============================================================================

feat: Added comprehensive unit testing and github action to run tests on new pull requests #8

Are you sure you want to change the base?

feat: Added comprehensive unit testing and github action to run tests on new pull requests #8

Uh oh!

Conversation

dklawren commented Jan 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgsheeh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgsheeh left a comment

Choose a reason for hiding this comment

Uh oh!

cgsheeh Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

cgsheeh Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

cgsheeh Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

cgsheeh Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

cgsheeh Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

cgsheeh Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants