Skip to content

use pypdl for download and handle chunk error#126

Merged
fqrious merged 2 commits into
mainfrom
use-pypdl-for-file-download
Jun 26, 2026
Merged

use pypdl for download and handle chunk error#126
fqrious merged 2 commits into
mainfrom
use-pypdl-for-file-download

Conversation

@fqrious

@fqrious fqrious commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@codecov

codecov Bot commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown

❌ 4 Tests Failed:

Tests completed Failed Passed Skipped
58 4 54 0
View the top 3 failed test(s) by shortest run time 
tests/unit/managers/test_base.py::test_create_relationship
Stack Traces | 0.001s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_cpe_name_as_dict_extracts_fields
Stack Traces | 0.001s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_parse_cpematch_date
Stack Traces | 0.001s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_split_cpe_name[cpe:2.3:a:apache:http_server:2.4.1:*:*:*:*:*:*:*-expected_split0]
Stack Traces | 0.001s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_split_cpe_name[cpe:2.3:a:microsoft:windows\\:server:2019:*:*:*:*:*:*:*-expected_split1]
Stack Traces | 0.001s run time 
No failure message available
tests/unit/cli/test_main.py::test_main
Stack Traces | 0.002s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_get_object_chunks
Stack Traces | 0.002s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_grouping_revoked[2-False]
Stack Traces | 0.002s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_parse_objects__null_matchdata
Stack Traces | 0.002s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_parse_software
Stack Traces | 0.002s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_parse_software_returns_valid_software
Stack Traces | 0.002s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_parse_deprecations
Stack Traces | 0.003s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_get_single_chunk[not_vulnerable+vulnerable]
Stack Traces | 0.004s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_grouping_revoked[0-False]
Stack Traces | 0.004s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_grouping_revoked[1-True]
Stack Traces | 0.004s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_parse_objects_for_criteria
Stack Traces | 0.004s run time 
No failure message available
tests/unit/cli/test_main.py::test_parse_bad_argument
Stack Traces | 0.005s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_get_single_chunk[not vulnerable]
Stack Traces | 0.005s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_get_single_chunk[vulnerable]
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_nvd.py::test_fetch_nvd_success
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_nvd.py::test_fetch_nvd_uses_api_key[None-None-None]
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_nvd.py::test_fetch_nvd_uses_api_key[None-env-env]
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_nvd.py::test_fetch_nvd_uses_api_key[arg-None-arg]
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_nvd.py::test_fetch_nvd_uses_api_key[arg-env-arg]
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_nvd.py::test_fetch_nvd_with_backoff
Stack Traces | 0.005s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_grouping__object_ref_contains_all_softwares
Stack Traces | 0.006s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_relate_indicator__vulnerable
Stack Traces | 0.006s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_title_db__lookup_raises_value_error_if_not_found
Stack Traces | 0.006s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_title_db__missing_lookup_calls_refresh
Stack Traces | 0.006s run time 
No failure message available
tests/unit/cli/test_main.py::test_parse_arguments[cli_args0-expected0]
Stack Traces | 0.007s run time 
No failure message available
tests/unit/managers/test_cve_cwe.py::test_relate_single
Stack Traces | 0.008s run time 
No failure message available
tests/unit/managers/test_base.py::test_do_process
Stack Traces | 0.009s run time 
No failure message available
tests/unit/managers/test_base.py::test_process
Stack Traces | 0.009s run time 
No failure message available
tests/unit/managers/test_cve_attack.py::test_relate_single
Stack Traces | 0.009s run time 
No failure message available
tests/unit/managers/test_cve_cwe.py::test_do_process
Stack Traces | 0.011s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_relate_indicator__not_vulnerable
Stack Traces | 0.02s run time 
No failure message available
tests/unit/managers/test_base.py::test_upload_edge_objects
Stack Traces | 0.023s run time 
No failure message available
tests/unit/managers/test_base.py::test_create_embedded_relationships
Stack Traces | 0.027s run time 
No failure message available
tests/unit/managers/test_base.py::test_get_edge_ids
Stack Traces | 0.032s run time 
No failure message available
tests/unit/managers/test_cve_attack.py::test_do_process
Stack Traces | 0.032s run time 
No failure message available
tests/unit/managers/test_base.py::test_upload_vertex_objects
Stack Traces | 0.069s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_title_db__lookup_swid_after_refresh
Stack Traces | 0.447s run time 
No failure message available
tests/unit/managers/test_cve_kev_cisa.py::test_relate_single
Stack Traces | 1.01s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_get_single_chunk[both vulnerable]
Stack Traces | 1.01s run time 
No failure message available
tests/unit/managers/test_cve_cwe.py::test_get_object_chunks
Stack Traces | 1.02s run time 
No failure message available
tests/unit/managers/test_cve_capec.py::test_relate_single
Stack Traces | 1.04s run time 
No failure message available
tests/unit/managers/test_cve_capec.py::test_do_process
Stack Traces | 1.07s run time 
No failure message available
tests/unit/tools/test_utils.py::test_import_default_objects
Stack Traces | 1.51s run time 
No failure message available
tests/unit/managers/test_cve_attack.py::test_get_object_chunks
Stack Traces | 3.13s run time 
No failure message available
tests/unit/managers/test_cve_kev_cisa.py::test_run_all
Stack Traces | 4.12s run time 
No failure message available
tests/unit/tools/test_cpe.py::test_refresh_from_api
Stack Traces | 45.6s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_get_cpematches__online
Stack Traces | 62.3s run time 
No failure message available
tests/unit/managers/test_cpematch.py::test_relate_single
Stack Traces | 575s run time 
No failure message available
View the full list of 5 ❄️ flaky test(s) 
tests/unit/cli/test_main.py::test_run_all

Flake rate in main: 13.51% (Passed 32 times, Failed 5 times)

Stack Traces | 2.68s run time 
acp_processor = <stix2arango.services.arangodb_service.ArangoDBService object at 0x7f8a18ffb650>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f8a18b858d0>

    def test_run_all(acp_processor, monkeypatch):
        with (
            patch(
                "arango_cve_processor.__main__.import_default_objects"
            ) as mock_import_defaults,
            patch("arango_cve_processor.__main__.create_indexes") as mock_create_indexes,
        ):
            processed_modes = []
            for v in RELATION_MANAGERS.values():
                monkeypatch.setattr(v, 'process', lambda self, *args, **kw: processed_modes.append(self.relationship_note))
>           run_all(
                acp_processor.db.name,
                modes=["cve-attack", 'cve-vulncheck-kev', "cve-capec"],
                start_date=date(2026, 1, 1),
                end_date=date(2025, 1, 1),
                modified_min="2999-01-01",
            )

.../unit/cli/test_main.py:61: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
arango_cve_processor/__main__.py:152: in run_all
    relation_manager = manager_klass(processor, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
arango_cve_processor/managers/cve_kev_vulncheck.py:31: in __init__
    self.verify_auth()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <arango_cve_processor.managers.cve_kev_vulncheck.VulnCheckKevManager object at 0x7f8a18d5d550>

    def verify_auth(self):
        resp = self.session.get("https://api.vulncheck.com/v3/index")
        if resp.status_code != 200:
>           raise ValueError(f"Bad API KEY for vulncheck: {resp.content}")
E           ValueError: Bad API KEY for vulncheck: b'{"error":true,"errors":["unauthorized"]}'

arango_cve_processor/managers/cve_kev_vulncheck.py:36: ValueError
tests/unit/managers/test_cve_capec.py::test_get_object_chunks

Flake rate in main: 8.11% (Passed 34 times, Failed 3 times)

Stack Traces | 2.08s run time 
No failure message available
tests/unit/managers/test_cve_kev_vulncheck.py::test_get_all_kevs

Flake rate in main: 18.42% (Passed 31 times, Failed 7 times)

Stack Traces | 1.33s run time 
acp_processor = <stix2arango.services.arangodb_service.ArangoDBService object at 0x7f8a18ffb650>

    @pytest.fixture
    def vulncheck_kev_manager(acp_processor):
>       manager = VulnCheckKevManager(acp_processor)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit/managers/test_cve_kev_vulncheck.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
arango_cve_processor/managers/cve_kev_vulncheck.py:31: in __init__
    self.verify_auth()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <arango_cve_processor.managers.cve_kev_vulncheck.VulnCheckKevManager object at 0x7f8a19a2d110>

    def verify_auth(self):
        resp = self.session.get("https://api.vulncheck.com/v3/index")
        if resp.status_code != 200:
>           raise ValueError(f"Bad API KEY for vulncheck: {resp.content}")
E           ValueError: Bad API KEY for vulncheck: b'{"error":true,"errors":["unauthorized"]}'

arango_cve_processor/managers/cve_kev_vulncheck.py:36: ValueError
tests/unit/managers/test_cve_kev_vulncheck.py::test_relate_single

Flake rate in main: 18.42% (Passed 31 times, Failed 7 times)

Stack Traces | 0.217s run time 
acp_processor = <stix2arango.services.arangodb_service.ArangoDBService object at 0x7f8a18ffb650>

    @pytest.fixture
    def vulncheck_kev_manager(acp_processor):
>       manager = VulnCheckKevManager(acp_processor)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit/managers/test_cve_kev_vulncheck.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
arango_cve_processor/managers/cve_kev_vulncheck.py:31: in __init__
    self.verify_auth()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <arango_cve_processor.managers.cve_kev_vulncheck.VulnCheckKevManager object at 0x7f8a18c41790>

    def verify_auth(self):
        resp = self.session.get("https://api.vulncheck.com/v3/index")
        if resp.status_code != 200:
>           raise ValueError(f"Bad API KEY for vulncheck: {resp.content}")
E           ValueError: Bad API KEY for vulncheck: b'{"error":true,"errors":["unauthorized"]}'

arango_cve_processor/managers/cve_kev_vulncheck.py:36: ValueError
tests/unit/managers/test_cve_kev_vulncheck.py::test_run_all

Flake rate in main: 21.05% (Passed 30 times, Failed 8 times)

Stack Traces | 0.278s run time 
acp_processor = <stix2arango.services.arangodb_service.ArangoDBService object at 0x7f8a18ffb650>

    @pytest.fixture
    def vulncheck_kev_manager(acp_processor):
>       manager = VulnCheckKevManager(acp_processor)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit/managers/test_cve_kev_vulncheck.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
arango_cve_processor/managers/cve_kev_vulncheck.py:31: in __init__
    self.verify_auth()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <arango_cve_processor.managers.cve_kev_vulncheck.VulnCheckKevManager object at 0x7f8a18b5b590>

    def verify_auth(self):
        resp = self.session.get("https://api.vulncheck.com/v3/index")
        if resp.status_code != 200:
>           raise ValueError(f"Bad API KEY for vulncheck: {resp.content}")
E           ValueError: Bad API KEY for vulncheck: b'{"error":true,"errors":["unauthorized"]}'

arango_cve_processor/managers/cve_kev_vulncheck.py:36: ValueError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@fqrious fqrious marked this pull request as ready for review June 26, 2026 16:23
@fqrious fqrious merged commit da0bda6 into main Jun 26, 2026
1 check failed
@fqrious fqrious deleted the use-pypdl-for-file-download branch June 26, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fqrious