Add exhaustive pytest test suite (363 tests) by deacon-mp · Pull Request #185 · mitre/training

deacon-mp · 2026-03-16T14:05:49Z

Summary

Add comprehensive pytest test suite with 363 tests covering the entire training plugin codebase
New tests/conftest.py with shared fixtures and framework stubs for standalone testing (no Caldera server needed)
Added pytest.ini for test configuration

Modules Covered

Core models: c_flag, c_badge, c_certification, c_exam, c_fillinblank, c_multiplechoice, c_navigator, errors
Services: base_flag (all static helpers), certificate_svc (signing, issuance, index), training_api (all endpoints)
Hook: enable, expansion, _load_flags, _apply_hidden_access
Flag modules (all categories):
- flags/advanced (flag_0, flag_1, flag_2)
- flags/adversaries (flag_0, flag_1, flag_2)
- flags/agents (flag_0-7, blue_0-3)
- flags/autonomous (blue_0, blue_1, blue_3)
- flags/operations (flag_0-3)
- flags/developers (flag_0, flag_2, flag_6, flag_7)
- flags/plugins (compass/flag_0, manx/flag_0-1, response/flag_0-1)
- flags/attack (blue_0, blue_1)
- flags/manual (blue_0, blue_1a)

Test plan

All 363 tests pass locally (pytest tests/ -v -- 0 failures)
Verify CI passes
Review test coverage for edge cases

Cover all modules: c_flag, c_badge, c_certification, c_exam, c_fillinblank, c_multiplechoice, c_navigator, errors, base_flag, certificate_svc, training_api, hook, and all flag submodules (advanced, adversaries, agents, autonomous, operations, developers, plugins/compass/manx/response, attack, manual). Includes shared conftest with framework stubs for standalone testing.

Copilot

Pull request overview

Adds an extensive pytest-based test suite for the training plugin, aiming to validate core models, services, hooks, and all flag categories without requiring a Caldera server runtime.

Changes:

Added many new unit tests (incl. TrainingApi, CertificateService, hook, models, and flag modules).
Introduced tests/conftest.py with stubs/fixtures to run tests standalone (mocking Caldera/framework deps).
Added pytest.ini to configure pytest discovery and asyncio behavior.

Reviewed changes

Copilot reviewed 26 out of 39 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_training_api.py	Adds broad coverage for `TrainingApi` endpoints/helpers.
tests/test_hook.py	Tests plugin hook entrypoints (`enable`, `expansion`, flag loading helpers).
tests/test_certificate_svc.py	Tests `CertificateService` signing/indexing/issuance logic with temp dirs.
tests/conftest.py	Provides shared fixtures and module stubs to run tests without Caldera/reportlab/aiohttp runtime.
tests/app/test_navigator.py	Tests `Navigator` flag behavior and helpers.
tests/app/test_multiplechoice.py	Tests `MultipleChoice` correctness, display contract, and multi-select.
tests/app/test_flag.py	Tests `Flag` base behavior (activation, completion timestamps, display, storage).
tests/app/test_fillinblank.py	Tests `FillInBlank` verification and display contract.
tests/app/test_exam.py	Tests `Exam` model display/unique/store behaviors.
tests/app/test_errors.py	Tests exception hierarchy/behavior for training plugin errors.
tests/app/test_certification.py	Expands `Certification` tests for badge/flag lookup, display, storage, unique.
tests/app/test_base_flag.py	Tests `BaseFlag` static helpers and standard verification/reset flows.
tests/app/test_badge.py	Expands `Badge` tests for display, unique, and multi-flag behavior.
tests/app/flags/plugins/test_flags.py	Adds tests for plugin flags (compass/manx/response).
tests/app/flags/operations/test_flags.py	Adds tests for operations flags 0–3.
tests/app/flags/manual/test_flags.py	Adds tests for manual blue flags.
tests/app/flags/developers/test_flags.py	Adds tests for developer flags.
tests/app/flags/autonomous/test_flags.py	Adds tests for autonomous blue flags.
tests/app/flags/attack/test_flags.py	Adds tests for ATT&CK quiz blue flags.
tests/app/flags/agents/test_flags.py	Adds tests for agent-related red flags.
tests/app/flags/agents/test_blue_flags.py	Adds tests for agent-related blue flags.
tests/app/flags/adversaries/test_flags.py	Adds tests for adversary-related flags.
tests/app/flags/advanced/test_flag2.py	Adds tests for advanced flag 2.
tests/app/flags/advanced/test_flag1.py	Adds tests for advanced flag 1.
tests/app/flags/advanced/test_flag0.py	Expands tests for advanced flag 0 (contact validation + verify).
pytest.ini	Configures pytest to discover tests and use asyncio auto mode.

Comments suppressed due to low confidence (4)

tests/test_training_api.py:1

AnswerFlag defines verify twice; the second (sync) method overrides the first (async) one. If any code path does await flag.verify(services) this will raise at runtime (awaiting a non-coroutine / bool). Use two distinct method names (e.g., keep async def verify(self, services) and add a separate def verify_answer(self, answer)), or use the real FillInBlank class in this test instead of overriding the contract of Flag.verify.
tests/conftest.py:1
conftest.py unconditionally injects an app module into sys.modules, which can silently override a real Caldera app package if it exists in the environment (e.g., running these tests from a larger Caldera checkout / monorepo). Prefer only stubbing when the import truly fails (try/except import), or gate this behavior behind an env var so the suite can run against real framework modules when available.
tests/conftest.py:1
This stub replaces aiohttp entirely, which can mask incompatibilities between the real aiohttp.web API and the plugin code (and can also break other test modules in the same run that rely on real aiohttp). If aiohttp is a manageable test dependency, consider using the real package; otherwise, keep the stub constrained (e.g., only patch the specific symbols used via monkeypatch in the tests that need it, rather than globally overriding sys.modules).
tests/app/test_flag.py:1
This assertion is a tautology and will always pass, so it doesn’t validate any behavior. If the intent is “doesn’t crash,” explicitly assert that the property returns a boolean; if the intent is to validate the missing-file behavior, patch the filesystem check (e.g., os.path.exists) to False and assert has_solution_guide is False.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/app/test_base_flag.py

+    async def test_correct_chain_length(self):
+        """Verifies the chain-length check independently."""
+        op = MagicMock()
+        op.chain = [MagicMock()]
+        services = {'data_svc': AsyncMock()}
+        services['data_svc'].locate = AsyncMock(return_value=[op])
+        # Patch the function to isolate the chain-length logic
+        with patch.object(BaseFlag, 'is_operation_successful', wraps=None) as mock_fn:
+            # Just test the underlying logic: chain length == num_links
+            assert len(op.chain) == 1
+
+    @pytest.mark.asyncio


- test_base_flag.py: rewrite test_correct_chain_length to actually call await BaseFlag.is_operation_successful() with mocked data_svc and a mock operation, asserting it returns True when chain length and traits match expectations

deacon-mp · 2026-03-18T13:18:40Z

@github-copilot review

deacon-mp requested a review from Copilot March 16, 2026 14:05

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Copilot started reviewing on behalf of deacon-mp March 16, 2026 14:31 View session

fix: address Copilot review feedback

7cd530f

- test_base_flag.py: rewrite test_correct_chain_length to actually call await BaseFlag.is_operation_successful() with mocked data_svc and a mock operation, asserting it returns True when chain length and traits match expectations

deacon-mp merged commit 1ec1c5f into master Mar 18, 2026
2 checks passed

deacon-mp deleted the test/exhaustive-pytest-coverage branch March 18, 2026 13:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add exhaustive pytest test suite (363 tests)#185

Add exhaustive pytest test suite (363 tests)#185
deacon-mp merged 2 commits intomasterfrom
test/exhaustive-pytest-coverage

deacon-mp commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

deacon-mp commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

deacon-mp commented Mar 16, 2026

Summary

Modules Covered

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

deacon-mp commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants