Skip to content

ScanCode tool not triggered for a coordinate - aircompressor #798

Description

@yashkohli88

The ScanCode tool is failing to trigger for the following coordinate:

git/github/airlift/aircompressor/e139a31be059855a5a70f6994a4ecaa0adbbea7a

Scancode Investigation Summary

Root Cause

  • Large binary test files (mozilla, samba, webster ~50-100MB each in testdata/silesia/)
  • 2 parallel scancode workers each loading files into memory
  • 1000s timeout allows full processing of these huge files

All Test Scenarios

# Scenario Timeout Processes Options Docker RAM Peak Memory Duration Files Completed Result
1 Original failure 1000s 2 - 7.65GB >7.65GB N/A 0 OOM SIGKILL
2 Manual docker exec 1000s 0 - 47GB ~17GB ~22 min 0/3 (all timed out) Success
3 Test: huge timeout 1000000s 1 --quiet 16GB >16GB N/A N/A OOM SIGKILL
4 Test: huge timeout 1000000s 1 --quiet 47GB ~36GB+ N/A N/A OOM SIGKILL
5 Test: --ignore patterns 1000000s 2 --ignore */testdata/* etc. 16GB ~2GB ~2 min N/A (skipped) Success (files excluded)
6 Test: default timeout 120s 2 - 47GB ~2GB ~5 min 0/3 (all timed out) Success
7 Final: original config 1000s 2 - 47GB 27.57GB ~33 min 2/3 (samba timed out) Success

Key Findings

  • Original config (timeout: 1000, processes: 2) needs ~28GB RAM
  • Original Docker limit was 7.65GB -> OOM
  • Manual run with processes: 0 (debug mode) uses less memory (~17GB) but slower
  • Default timeout (120s) skips large test files, only needs ~2GB
  • --ignore patterns can exclude problematic files entirely (~2GB, fastest)
  • Huge timeout (1000000s) forces full binary parsing → 36GB+ → OOM even at 47GB

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions