Implement Smart File Type Detection and Filtering

[Kcodess2807]

Problem Description

Current filtering relies only on file extensions and patterns. This misses files without extensions, incorrectly categorizes files, and doesn't support content-based filtering.

Current Behavior

# Only extension-based filtering
filters = FilterCriteria(
    file_extensions={".py", ".js"},  # Misses Python files without .py extension
    exclude_patterns=["*.log"]       # Misses log files with different extensions
)

Desired Behavior

# Smart content-based filtering
filters = FilterCriteria(
    file_types=["python", "javascript", "config"],  # Detect by content
    exclude_file_types=["binary", "log", "cache"],  # Smart exclusion
    content_patterns=["#!/usr/bin/env python"],     # Shebang detection
    max_binary_size=1024                            # Skip large binaries
)

[Einswilli]

This feature is also implemented. Please refer to the forklet.core.filter.FilterEngine and the forklet.models.downloads.FilterCriteria model for details.

However, implementing content-based filtering would negatively impact performance. It would slow down the download process and consume additional API requests, leading to rate limit concerns. This is because the GitHub API used by Forklet does not support content-based filtering. We would be forced to download all files first—including their base64-encoded content—and then perform the filtering locally, which is an inefficient process.

[Kcodess2807]

Thank you for the detailed explanation, @Einswilli You raise valid performance concerns about content-based filtering.

What if we implement metadata-based smart detection without downloading content?

filters = FilterCriteria(
    # Use filename heuristics (no content download needed)
    detect_shebang_from_name=True,      # Detect scripts by naming patterns
    smart_extension_mapping={
        "Dockerfile": "config",
        "Makefile": "build",
        ".gitignore": "config"
    },
    
    # Enhanced pattern matching
    extensionless_file_patterns=[
        "LICENSE*", "README*", "Dockerfile*", "Makefile*"
    ],
    
    # Size-based binary detection (from API metadata)
    auto_exclude_large_files=True,
    large_file_threshold=10_000_000     # 10MB
)

Benefits:

• No additional API calls
• Uses existing GitHub tree API metadata (path, size, type)
• Fast filtering based on naming conventions
• Extensible mapping system