Skip to content

v2.5.0-beta.1 #729

Merged
whikernel merged 1110 commits intomasterfrom
develop
Feb 27, 2025
Merged

v2.5.0-beta.1 #729
whikernel merged 1110 commits intomasterfrom
develop

Conversation

@whikernel
Copy link
Contributor

@whikernel whikernel commented Feb 26, 2025

This pull request introduces several significant changes to the backend, development environment setup, continuous integration workflow. The most important changes include the migration partial migration of the API to a REST API v2.
A restructuring of the backend code into business is also ongoing. The old API should still works but bugs are to be expected hence the beta tag. Bug and security patches are also included.

This is a BETA version and should not be put in production

API

  • Introduction of REST API v2
  • Migration to a more structured backend

Development Environment Setup:

  • Added configuration files for development containers, including Dockerfile, devcontainer.json, and docker-compose.yml, to streamline the development environment setup.

Continuous Integration Workflow:

  • Updated the GitHub Actions workflow to include separate jobs for static analysis, building Docker images, generating GraphQL documentation, and running tests.

Configuration Updates:

  • Added new environment variables and configuration options in .env.model to support additional features and services.

Summary by CodeRabbit

  • New Features

    • Updated product version to 2.5.0‑beta.1.
    • Introduced a comprehensive set of RESTful API endpoints (versioned under /api/v2) for managing cases, assets, IOCs, tasks, dashboards, search, and user profiles—enabling enhanced data export and filtering.
    • Added new real‑time collaboration and notification capabilities via enhanced socket event handlers.
  • Improvements

    • Streamlined developer workflows with updated Docker, CI pipelines, and environment configuration.
    • Refined authentication and UI elements for login and profile management for better user experience.

if not next_url or urlsplit(next_url).netloc != '':
next_url = url_for('index.index', cid=user.ctx_case)

return redirect(next_url) No newline at end of file

Check warning

Code scanning / CodeQL

URL redirection from remote source Medium

Untrusted URL redirection depends on a
user-provided value
.

Copilot Autofix

AI 12 months ago

To fix the problem, we need to enhance the validation of the next_url parameter to ensure it is safe for redirection. We can achieve this by:

  1. Improving the is_safe_url function to handle additional edge cases, such as URLs with backslashes and mistyped URLs.
  2. Ensuring that the next_url is either a relative URL or has the same host as the current request.

We will modify the is_safe_url function to include these additional checks and update the _filter_next_url function to use the improved validation.

Suggested changeset 1
source/app/business/auth.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/source/app/business/auth.py b/source/app/business/auth.py
--- a/source/app/business/auth.py
+++ b/source/app/business/auth.py
@@ -81,7 +81,8 @@
     Check whether the target URL is safe for redirection by ensuring that it is either a relative URL or
-    has the same host as the current request.
+    has the same host as the current request. Also handles backslashes and mistyped URLs.
     """
+    target = target.replace('\\', '')
     ref_url = urlparse(request.host_url)
     test_url = urlparse(urljoin(request.host_url, target))
-    return test_url.scheme in ('http', 'https') and ref_url.netloc == test_url.netloc
+    return test_url.scheme in ('http', 'https') and ref_url.netloc == test_url.netloc and not test_url.path.startswith('//')
 
@@ -94,4 +95,2 @@
         return url_for('index.index', cid=context_case)
-    # Remove backslashes to mitigate obfuscation
-    next_url = next_url.replace('\\', '')
     if is_safe_url(next_url):
EOF
@@ -81,7 +81,8 @@
Check whether the target URL is safe for redirection by ensuring that it is either a relative URL or
has the same host as the current request.
has the same host as the current request. Also handles backslashes and mistyped URLs.
"""
target = target.replace('\\', '')
ref_url = urlparse(request.host_url)
test_url = urlparse(urljoin(request.host_url, target))
return test_url.scheme in ('http', 'https') and ref_url.netloc == test_url.netloc
return test_url.scheme in ('http', 'https') and ref_url.netloc == test_url.netloc and not test_url.path.startswith('//')

@@ -94,4 +95,2 @@
return url_for('index.index', cid=context_case)
# Remove backslashes to mitigate obfuscation
next_url = next_url.replace('\\', '')
if is_safe_url(next_url):
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
@coderabbitai
Copy link

coderabbitai bot commented Feb 26, 2025

Walkthrough

This pull request introduces extensive project improvements across every layer. The version was bumped in the configuration, and new files were added for containerization, Docker Compose, Kubernetes deployments, and CI/CD pipelines. E2E testing support and documentation (including code style and architecture) were enhanced. Meanwhile, a broad refactoring of the Flask back‑end occurred: routes and blueprints (both traditional and REST‑v2) were renamed, reorganized, and deprecated or removed where appropriate. In addition, several business logic modules have been updated for consistent naming and improved error handling, and new Socket.IO event handlers were added for real‑time interactions.

Changes

Files (grouped) Change Summary
Version & Config
.bumpversion.cfg
Updated current_version from 2.4.18 to 2.5.0‑beta.1 indicating a switch toward beta versioning.
Container & Deployment
.devcontainer/*, docker-compose.base.yml, docker-compose.dev.yml, docker/nginx/*, docker/webApp/*, deploy/kubernetes/charts/*, deploy/kubernetes/charts/values.yaml
Added development container and Docker Compose files; reformatted Docker commands; introduced new Docker‑build stages and updated Kubernetes Helm chart templates with added security contexts and resource requests.
CI/CD, Code Style & Documentation
.github/workflows/ci.yml, CODESTYLE.md, README.md, architecture.md
Overhauled the GitHub Actions workflow with new granular jobs; restructured code style guidelines; updated documentation including logo path and comprehensive architectural overview.
E2E Testing Infrastructure
e2e/.gitignore, e2e/data/report.md, e2e/package.json, e2e/playwright.config.js, e2e/tests/**
Introduced a new E2E testing framework with Playwright configuration, end‑to‑end test scripts, and reporting capabilities.
Application Backend & Blueprints
source/app/__init__.py, source/app/alembic/**, source/app/blueprints/** (including deleted files such as case_assets_routes.py, case_ioc_routes.py, etc.)
Major refactoring of blueprints and routes, including renaming functions for consistency (e.g. prefixing with “cases_”, “ioc_”, “notes_”), removal or deprecation of obsolete routes, and improvements in access control and error responses.
Socket.IO Event Handlers
source/app/blueprints/socket_io_event_handlers/**
Added new Socket.IO event handlers for case summary and case notes events, plus a collaboration notification handler with access control.
Business Logic Modules
source/app/business/assets.py, auth.py, cases.py, errors.py, iocs.py, notes.py
Refactored business logic: functions were renamed (e.g. to “assets_create”, “cases_create”, “ioc_create”, “notes_update”), error handling was standardized (introducing ObjectNotFoundError), and consistency improved across data‐processing functionalities.
REST API v2 Implementation
source/app/blueprints/rest/v2/**
Introduced a new, versioned REST API layer with organized blueprints covering authentication, cases, assets, IOCs, tasks, dashboard, context, alerts and more, enabling a modern and modular API structure.

Sequence Diagram(s)

sequenceDiagram
    participant C as Client
    participant API as REST API (v2)
    participant BL as Business Logic
    participant DB as Database
    C->>API: GET /api/v2/cases/assets?case_identifier=X
    API->>BL: Retrieve asset details (assets_get_detailed)
    BL->>DB: Query asset record for case X
    DB-->>BL: Asset data returned
    BL->>API: Format results via schema
    API-->>C: 200 OK with asset data
Loading
sequenceDiagram
    participant U as User
    participant S as Socket.IO Server
    U->>S: Emit "join-case-obj-notif" with channel info
    S->>S: join_room(channel)
    S-->>U: Emit confirmation notification (user joined)
Loading

Poem

I'm a bouncy little rabbit in a code field so bright,
Hopping through version bumps and deployments in flight.
My ears perk up at Docker and Kubernetes galore,
As CI/CD pipelines and REST APIs open up the door.
With playful hops through Socket events and tests that delight,
I celebrate refactors and updates with pure bunny might! 🐰✨

✨ Finishing Touches
  • 📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

coderabbitai[bot]

This comment was marked as off-topic.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (4)
source/app/business/auth.py (4)

17-29: Consider annotating the return type.

Currently, this function returns either a user object or None, but the signature does not reflect this. Adding an explicit return type (e.g., -> Optional[User]) can improve code clarity and maintainability.

-def _retrieve_user_by_username(username:str):
+def _retrieve_user_by_username(username: str) -> Optional[User]:
     ...

30-56: Align the docstring with the actual return type.

The docstring states it returns a “User object,” but it actually returns a serialized dictionary from UserSchema. Consider updating the docstring for accuracy. Also, to enhance debugging, you may use log.exception(e) instead of log.error(e.__str__()) to capture the stack trace.

     :return: User object if successful, None otherwise
+    #:return: Dictionary if successful, None otherwise

- log.error(e.__str__())
+ log.exception("Failed to validate LDAP login:")

58-77: Keep “validate” functions pure by removing session-altering side effects.

Unlike validate_ldap_login, this function calls wrap_login_user, causing a side effect that creates a session. For consistency and improved separation of concerns, either centralize session handling in the caller or unify both validation flows to maintain a single approach.


109-110: Simplify nested conditions.

Ruff suggests combining nested checks into a single if. This can make the code more concise and reduce indentation.

-if app.config['SERVER_SETTINGS']['enforce_mfa'] is True and is_oidc is False:
-    if "mfa_verified" not in session or session["mfa_verified"] is False:
+if (
+    app.config['SERVER_SETTINGS']['enforce_mfa']
+    and not is_oidc
+    and (
+        "mfa_verified" not in session
+        or session["mfa_verified"] is False
+    )
):
     return redirect(url_for('mfa_verify'))
🧰 Tools
🪛 Ruff (0.8.2)

109-110: Use a single if statement instead of nested if statements

(SIM102)

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ccec2b7 and 83c9ffe.

📒 Files selected for processing (1)
  • source/app/business/auth.py (1 hunks)
🧰 Additional context used
🪛 Ruff (0.8.2)
source/app/business/auth.py

109-110: Use a single if statement instead of nested if statements

(SIM102)

🪛 GitHub Check: CodeQL
source/app/business/auth.py

[warning] 134-134: URL redirection from remote source
Untrusted URL redirection depends on a user-provided value.

⏰ Context from checks skipped due to timeout of 90000ms (1)
  • GitHub Check: End to end tests
🔇 Additional comments (3)
source/app/business/auth.py (3)

79-87: Verify edge cases under proxies or subdomains.

While this function helps prevent open redirects by matching the netloc, it might block legitimate subdomain redirects if your deployment uses them or if requests come through proxies with rewritten host headers. Verify that this meets your architectural needs.


88-100: Looks good for preventing open redirects.

Your _filter_next_url function effectively ensures that untrusted or malformed URLs are discarded in favor of your site’s index page. This is a robust pattern for handling next URLs.


134-134: ** Existing concern about untrusted URL redirection.**

GitHub advanced security flagged this line for untrusted redirection. However, _filter_next_url checks domain safety and defaults to the index page. This mitigates open redirect risks unless subdomains or special proxy headers require broader matching.

🧰 Tools
🪛 GitHub Check: CodeQL

[warning] 134-134: URL redirection from remote source
Untrusted URL redirection depends on a user-provided value.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
README.md (1)

3-3: Add Alternate Text to the Logo Image

The updated image tag on this line no longer includes an alt attribute, which is required for accessibility and to satisfy markdownlint (MD045). Please include descriptive alternate text (e.g., "IRIS Logo") to improve accessibility.

-    <img src="ui/public/assets/img/logo.ico" />
+    <img src="ui/public/assets/img/logo.ico" alt="IRIS Logo" />
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)

3-3: Images should have alternate text (alt text)
null

(MD045, no-alt-text)

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 83c9ffe and 9441d4d.

📒 Files selected for processing (3)
  • README.md (1 hunks)
  • source/app/blueprints/graphql/graphql_route.py (2 hunks)
  • source/app/blueprints/rest/alerts_routes.py (25 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • source/app/blueprints/graphql/graphql_route.py
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
README.md

3-3: Images should have alternate text (alt text)
null

(MD045, no-alt-text)

⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: End to end tests
  • GitHub Check: Generate graphQL documentation
  • GitHub Check: Test API
  • GitHub Check: Build docker app
🔇 Additional comments (8)
README.md (1)

9-9: Ensure Version Consistency in Documentation

The “Current Version” displayed is still set to v2.4.20, which appears inconsistent with the beta release identified as v2.5.0-beta.1. For clarity and to prevent confusion among users, please update all version references accordingly (e.g., the version text and any related git checkout instructions).

-  <i>Current Version v2.4.20</i>
+  <i>Current Version v2.5.0-beta.1</i>
source/app/blueprints/rest/alerts_routes.py (7)

66-66: Blueprint renaming for REST API v2 transition.

The blueprint name change from alerts_blueprint to alerts_rest_blueprint aligns with the PR objective of transitioning to REST API v2. This is part of the broader backend restructuring mentioned in the PR summary.


69-71: API versioning and deprecation strategy implemented.

Good implementation of API versioning. The endpoint_deprecated decorator properly marks old endpoints while pointing to their new v2 counterparts.


159-159: Enhanced security with user context in database queries.

The addition of current_user_id parameter to the get_filtered_alerts function is a good security practice as it ensures database queries are scoped to the current user's permissions.


169-178: Improved error handling for schema field selection.

The addition of try-except block with logging for schema field selection provides better error handling. If invalid fields are specified, the code gracefully falls back to using the full schema rather than failing.


835-835: More robust identifier parsing.

Using parse_comma_separated_identifiers() provides a more robust way to handle multiple identifiers compared to the previous manual string splitting approach.

Also applies to: 904-904


416-420: Code clarity improvement in activity tracking.

The restructured activity tracking and history entry code makes the purpose clearer by separating the tracking activity from adding the history entry.


217-218:

Details

❓ Verification inconclusive

Note the change in validation behavior.

The addition of partial=True parameter to schema loading changes validation behavior. This allows partial objects to be validated without requiring all fields, which makes the API more flexible but could potentially allow incomplete data if not handled properly.


🏁 Script executed:

#!/bin/bash
# Look for other places where these schemas are used with partial=True to ensure consistent behavior
rg "ioc_schema.load|asset_schema.load" -A 3 -B 3

Length of output: 5434


Attention: Validate Impact of Partial Validation Change

The new use of partial=True in loading iocs and assets allows the API to accept partial data. This change is intentional and aligns with similar patterns elsewhere in the codebase (e.g., in source/app/business/iocs.py). However, please verify that downstream processing correctly handles incomplete data, ensuring that missing fields are either safely optional or explicitly managed.

  • Confirm that alert processing (including deserialization in alerts_routes.py) is robust against partial data.
  • Review downstream dependencies and tests to ensure that accepting incomplete objects does not introduce issues.

@whikernel whikernel merged commit a4bfeda into master Feb 27, 2025
30 of 31 checks passed
@coderabbitai coderabbitai bot mentioned this pull request Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Comments