Merge branch 'main' into fix-docker-env-secrets

Author: paarthbhatt

README.rst

@@ -19,10 +19,10 @@ VulnerableCode
 
 
 VulnerableCode is a free and open database of open source software package
-vulnerabilities **because open source software vulnerabilities data and tools
+vulnerabilities **because open source software vulnerability data and tools
 should be free and open source themselves**:
 
-we are trying to change this and evolve the status quo in a few other areas!
+We are trying to change this and evolve the status quo in a few other areas!
 
 - Vulnerability databases have been **traditionally proprietary** even though they
   are mostly about free and open source software. 
@@ -31,13 +31,13 @@ we are trying to change this and evolve the status quo in a few other areas!
   means a lot of false positive signals that require extensive expert reviews.
 
 - Vulnerability databases are also mostly about vulnerabilities first and software
-  package second, making it difficult to find if and when a vulnerability applies
-  to a piece of code. VulnerableCode focus is on software package first where
-  a Package URL is a key and natural identifier for packages; this is making it
+  packages second, making it difficult to find if and when a vulnerability applies
+  to a piece of code. VulnerableCode's focus is on software packages first where
+  a Package URL (PURL) is a key and natural identifier for packages; this makes it
   easier to find a package and whether it is vulnerable.
 
-Package URL themselves were designed first in ScanCode and VulnerableCode
-and are now a de-facto standard for vulnerability management and package references.
+PURLs were designed initially for ScanCode and VulnerableCode. PURL is
+now a de-facto standard for vulnerability management and package references.
 See https://github.com/package-url/purl-spec
 
 The VulnerableCode project is a FOSS community resource to help improve the
@@ -49,17 +49,14 @@ the database current.
 
 .. pull-quote::
    **Warning**
+   VulnerableCode is under active development and may not be ready for production
+use depending on your use cases.
 
-   VulnerableCode is under active development and is not yet fully
-   usable.
+Read more about VulnerableCode at https://vulnerablecode.readthedocs.org/
 
-
-Read more about VulnerableCode https://vulnerablecode.readthedocs.org/
-
-VulnerableCode tech stack is Python, Django, PostgreSQL, nginx and Docker and
+The VulnerableCode tech stack is Python, Django, PostgreSQL, nginx and Docker and
 several libraries.
 
-
 Getting started
 ===============
 

docs/source/contributing.rst

@@ -17,7 +17,7 @@ resources to help you get started.
 Do Your Homework
 ----------------
 
-Before adding a contribution or create a new issue, take a look at the project’s
+Before adding a contribution or create a new issue, take a look at the project's
 `README <https://github.com/aboutcode-org/vulnerablecode>`_, read through our
 `documentation <https://vulnerablecode.readthedocs.io/en/latest/>`_,
 and browse existing `issues <https://github.com/aboutcode-org/vulnerablecode/issues>`_,
@@ -73,7 +73,7 @@ overlooked. We value any suggestions to improve
 
 .. tip::
     Our documentation is treated like code. Make sure to check our
-    `writing guidelines <https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_doc.html>`_
+    `writing guidelines <https://scancode-toolkit.readthedocs.io/en/stable/getting-started/contribute/contributing-docs.html>`_
     to help guide new users.
 
 Other Ways
@@ -87,7 +87,7 @@ questions, and interact with us and other community members on
 Helpful Resources
 -----------------
 
-- Review our `comprehensive guide <https://scancode-toolkit.readthedocs.io/en/latest/contribute/index.html>`_
+- Review our `comprehensive guide <https://scancode-toolkit.readthedocs.io/en/stable/getting-started/contribute/index.html>`_
   for more details on how to add quality contributions to our codebase and documentation
 - Check this free resource on `How to contribute to an open source project on github <https://egghead.io/lessons/javascript-identifying-how-to-contribute-to-an-open-source-project-on-github>`_
 - Follow `this wiki page <https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>`_

docs/source/tutorial_add_importer_pipeline.rst

@@ -152,7 +152,7 @@ At this point, an example importer will look like this:
 .. code-block:: python
     :caption: vulnerabilities/pipelines/example_importer.py
     :linenos:
-    :emphasize-lines: 16-17, 20-21, 23-24
+    :emphasize-lines: 17-18, 21-22, 24-25
 
     from vulnerabilities.pipelines import VulnerableCodeBaseImporterPipeline
 
@@ -165,6 +165,7 @@ At this point, an example importer will look like this:
         license_url = "https://exmaple.org/license/"
         spdx_license_expression = "CC-BY-4.0"
         importer_name = "Example Importer"
+        run_once = False
 
         @classmethod
         def steps(cls):
@@ -196,7 +197,7 @@ version management from `univers <https://github.com/aboutcode-org/univers>`_.
 .. code-block:: python
     :caption: vulnerabilities/pipelines/example_importer.py
     :linenos:
-    :emphasize-lines: 34-35, 37-40
+    :emphasize-lines: 35-36, 38-41
 
     from datetime import datetime
     from datetime import timezone
@@ -223,6 +224,7 @@ version management from `univers <https://github.com/aboutcode-org/univers>`_.
         license_url = "https://example.org/license/"
         spdx_license_expression = "CC-BY-4.0"
         importer_name = "Example Importer"
+        run_once = False
 
         @classmethod
         def steps(cls):
@@ -303,6 +305,17 @@ version management from `univers <https://github.com/aboutcode-org/univers>`_.
    Implement ``on_failure`` to handle cleanup in case of pipeline failure.
    Cleanup of downloaded archives or cloned repos is necessary to avoid potential resource leakage.
 
+.. tip::
+
+   Set ``run_once`` to ``True`` if pipeline is meant to be run once.
+
+   - To rerun onetime pipeline, reset ``is_active`` to ``True`` via a migration, pipeline will
+     run one more time and then deactivate.
+
+   - To convert a onetime pipeline to a regular pipeline, set the ``run_once`` class variable
+     to ``False`` and reset ``is_active` field to ``True`` via a migration.
+
+
 .. note::
 
    | Use ``make valid`` to format your new code using black and isort automatically.

vulnerabilities/importer.py

@@ -512,24 +512,30 @@ def from_dict(cls, affected_pkg: dict):
         fixed_version_range = None
         affected_range = affected_pkg["affected_version_range"]
         fixed_range = affected_pkg["fixed_version_range"]
-        introduced_by_commit_patches = (
-            affected_pkg.get("introduced_by_package_commit_patches") or []
-        )
-        fixed_by_commit_patches = affected_pkg.get("fixed_by_package_commit_patches") or []
+        introduced_by_commit_patches = affected_pkg.get("introduced_by_commit_patches") or []
+        fixed_by_commit_patches = affected_pkg.get("fixed_by_commit_patches") or []
 
         try:
-            affected_version_range = VersionRange.from_string(affected_range)
-            fixed_version_range = VersionRange.from_string(fixed_range)
+            affected_version_range = (
+                VersionRange.from_string(affected_range) if affected_range else None
+            )
+            fixed_version_range = VersionRange.from_string(fixed_range) if fixed_range else None
         except:
             tb = traceback.format_exc()
             logger.error(
                 f"Cannot create AffectedPackage with invalid or unknown range: {affected_pkg!r} with error: {tb!r}"
             )
             return
 
-        if not fixed_version_range and not affected_version_range:
+        if (
+            not fixed_version_range
+            and not affected_version_range
+            and not introduced_by_commit_patches
+            and not fixed_by_commit_patches
+        ):
             logger.error(
-                f"Cannot create AffectedPackage without fixed or affected range: {affected_pkg!r}"
+                f"Cannot create an AffectedPackage for: {affected_pkg!r}, at least one of the following must be provided: "
+                "a fixed version range, an affected version range, introduced commit patches, or fixed commit patches"
             )
             return
 
@@ -575,6 +581,10 @@ class AdvisoryData:
     original_advisory_text: Optional[str] = None
 
     def __post_init__(self):
+        if self.advisory_id and self.advisory_id in self.aliases:
+            raise ValueError(
+                f"advisory_id {self.advisory_id} should not be present in aliases {self.aliases}"
+            )
         if self.summary:
             self.summary = self.clean_summary(self.summary)
 

vulnerabilities/importers/__init__.py

@@ -55,10 +55,18 @@
 from vulnerabilities.pipelines.v2_importers import istio_importer as istio_importer_v2
 from vulnerabilities.pipelines.v2_importers import mattermost_importer as mattermost_importer_v2
 from vulnerabilities.pipelines.v2_importers import mozilla_importer as mozilla_importer_v2
+from vulnerabilities.pipelines.v2_importers import nginx_importer as nginx_importer_v2
 from vulnerabilities.pipelines.v2_importers import npm_importer as npm_importer_v2
 from vulnerabilities.pipelines.v2_importers import nvd_importer as nvd_importer_v2
+from vulnerabilities.pipelines.v2_importers import openssl_importer as openssl_importer_v2
 from vulnerabilities.pipelines.v2_importers import oss_fuzz as oss_fuzz_v2
 from vulnerabilities.pipelines.v2_importers import postgresql_importer as postgresql_importer_v2
+from vulnerabilities.pipelines.v2_importers import (
+    project_kb_msr2019_importer as project_kb_msr2019_importer_v2,
+)
+from vulnerabilities.pipelines.v2_importers import (
+    project_kb_statements_importer as project_kb_statements_importer_v2,
+)
 from vulnerabilities.pipelines.v2_importers import pypa_importer as pypa_importer_v2
 from vulnerabilities.pipelines.v2_importers import pysec_importer as pysec_importer_v2
 from vulnerabilities.pipelines.v2_importers import redhat_importer as redhat_importer_v2
@@ -87,8 +95,11 @@
         github_osv_importer_v2.GithubOSVImporterPipeline,
         redhat_importer_v2.RedHatImporterPipeline,
         aosp_importer_v2.AospImporterPipeline,
+        project_kb_statements_importer_v2.ProjectKBStatementsPipeline,
+        project_kb_msr2019_importer_v2.ProjectKBMSR2019Pipeline,
         ruby_importer_v2.RubyImporterPipeline,
         epss_importer_v2.EPSSImporterPipeline,
+        nginx_importer_v2.NginxImporterPipeline,
         mattermost_importer_v2.MattermostImporterPipeline,
         nvd_importer.NVDImporterPipeline,
         github_importer.GitHubAPIImporterPipeline,
@@ -109,6 +120,7 @@
         ruby.RubyImporter,
         apache_kafka.ApacheKafkaImporter,
         openssl.OpensslImporter,
+        openssl_importer_v2.OpenSSLImporterPipeline,
         redhat.RedhatImporter,
         archlinux.ArchlinuxImporter,
         ubuntu.UbuntuImporter,

vulnerabilities/management/commands/run_scheduler.py

@@ -17,16 +17,21 @@
 
 
 def init_pipeline_scheduled():
-    """Initialize schedule jobs for active PipelineSchedule."""
-    active_pipeline_qs = models.PipelineSchedule.objects.filter(is_active=True).order_by(
-        "created_date"
-    )
-    for pipeline_schedule in active_pipeline_qs:
-        if scheduled_job_exists(pipeline_schedule.schedule_work_id):
-            continue
-        new_id = pipeline_schedule.create_new_job()
-        pipeline_schedule.schedule_work_id = new_id
-        pipeline_schedule.save(update_fields=["schedule_work_id"])
+    """
+    Initialize schedule jobs for active PipelineSchedule.
+        - Create new schedule if there is no schedule for active pipeline
+        - Create new schedule if schedule is corrupted for an active pipeline
+        - Delete schedule for inactive pipeline
+    """
+    pipeline_qs = models.PipelineSchedule.objects.order_by("created_date")
+    for pipeline in pipeline_qs:
+        reset_schedule = pipeline.is_active != bool(pipeline.schedule_work_id)
+        if not scheduled_job_exists(pipeline.schedule_work_id):
+            reset_schedule = True
+
+        if reset_schedule:
+            pipeline.schedule_work_id = pipeline.create_new_job()
+            pipeline.save(update_fields=["schedule_work_id"])
 
 
 class Command(rqscheduler.Command):

vulnerabilities/migrations/0110_pipelineschedule_is_run_once.py

@@ -0,0 +1,22 @@
+# Generated by Django 4.2.22 on 2026-01-08 13:41
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("vulnerabilities", "0109_alter_advisoryseverity_scoring_elements_and_more"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="pipelineschedule",
+            name="is_run_once",
+            field=models.BooleanField(
+                db_index=True,
+                default=False,
+                help_text="When set to True, this Pipeline will run only once.",
+            ),
+        ),
+    ]

vulnerabilities/migrations/0111_alter_advisoryseverity_scoring_system_and_more.py

@@ -0,0 +1,61 @@
+# Generated by Django 4.2.25 on 2026-01-19 06:22
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("vulnerabilities", "0110_pipelineschedule_is_run_once"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="advisoryseverity",
+            name="scoring_system",
+            field=models.CharField(
+                choices=[
+                    ("cvssv2", "CVSSv2 Base Score"),
+                    ("cvssv3", "CVSSv3 Base Score"),
+                    ("cvssv3.1", "CVSSv3.1 Base Score"),
+                    ("cvssv4", "CVSSv4 Base Score"),
+                    ("rhbs", "RedHat Bugzilla severity"),
+                    ("rhas", "RedHat Aggregate severity"),
+                    ("archlinux", "Archlinux Vulnerability Group Severity"),
+                    ("cvssv3.1_qr", "CVSSv3.1 Qualitative Severity Rating"),
+                    ("generic_textual", "Generic textual severity rating"),
+                    ("apache_httpd", "Apache Httpd Severity"),
+                    ("apache_tomcat", "Apache Tomcat Severity"),
+                    ("epss", "Exploit Prediction Scoring System"),
+                    ("ssvc", "Stakeholder-Specific Vulnerability Categorization"),
+                    ("openssl", "OpenSSL Severity"),
+                ],
+                help_text="Identifier for the scoring system used. Available choices are: cvssv2: CVSSv2 Base Score,\ncvssv3: CVSSv3 Base Score,\ncvssv3.1: CVSSv3.1 Base Score,\ncvssv4: CVSSv4 Base Score,\nrhbs: RedHat Bugzilla severity,\nrhas: RedHat Aggregate severity,\narchlinux: Archlinux Vulnerability Group Severity,\ncvssv3.1_qr: CVSSv3.1 Qualitative Severity Rating,\ngeneric_textual: Generic textual severity rating,\napache_httpd: Apache Httpd Severity,\napache_tomcat: Apache Tomcat Severity,\nepss: Exploit Prediction Scoring System,\nssvc: Stakeholder-Specific Vulnerability Categorization,\nopenssl: OpenSSL Severity ",
+                max_length=50,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="vulnerabilityseverity",
+            name="scoring_system",
+            field=models.CharField(
+                choices=[
+                    ("cvssv2", "CVSSv2 Base Score"),
+                    ("cvssv3", "CVSSv3 Base Score"),
+                    ("cvssv3.1", "CVSSv3.1 Base Score"),
+                    ("cvssv4", "CVSSv4 Base Score"),
+                    ("rhbs", "RedHat Bugzilla severity"),
+                    ("rhas", "RedHat Aggregate severity"),
+                    ("archlinux", "Archlinux Vulnerability Group Severity"),
+                    ("cvssv3.1_qr", "CVSSv3.1 Qualitative Severity Rating"),
+                    ("generic_textual", "Generic textual severity rating"),
+                    ("apache_httpd", "Apache Httpd Severity"),
+                    ("apache_tomcat", "Apache Tomcat Severity"),
+                    ("epss", "Exploit Prediction Scoring System"),
+                    ("ssvc", "Stakeholder-Specific Vulnerability Categorization"),
+                    ("openssl", "OpenSSL Severity"),
+                ],
+                help_text="Identifier for the scoring system used. Available choices are: cvssv2: CVSSv2 Base Score,\ncvssv3: CVSSv3 Base Score,\ncvssv3.1: CVSSv3.1 Base Score,\ncvssv4: CVSSv4 Base Score,\nrhbs: RedHat Bugzilla severity,\nrhas: RedHat Aggregate severity,\narchlinux: Archlinux Vulnerability Group Severity,\ncvssv3.1_qr: CVSSv3.1 Qualitative Severity Rating,\ngeneric_textual: Generic textual severity rating,\napache_httpd: Apache Httpd Severity,\napache_tomcat: Apache Tomcat Severity,\nepss: Exploit Prediction Scoring System,\nssvc: Stakeholder-Specific Vulnerability Categorization,\nopenssl: OpenSSL Severity ",
+                max_length=50,
+            ),
+        ),
+    ]