feat: add --exclude flag to skip paths during scanning

[Ankitsinghsisodya]

Overview

Issue: #2324

This PR adds a --exclude CLI flag that allows users to specify glob patterns to exclude files and directories from vulnerability scanning.

Fixes #2324

Details

Problem

Users scanning large repositories often want to exclude test files, documentation, and other non-production code from vulnerability scans. While .gitignore is respected by default and osv-scanner.toml can be used, there was no quick CLI option to exclude paths.

Solution

Added --exclude / -e flag that accepts glob patterns:

osv-scanner scan source -r --exclude="**/test/**" --exclude="**/docs/**" ./path

[another-rex]

This is definitely something we want, though might need a bit more design before including it. It is really hard for us to justify removing a flag once it's added.

I want a way for people to specify regex patterns without having to add another separate flag. Maybe something like if it is wrapped /<regex>/ like in javascript?

[G-Rath]

Appreciate your work on this!

Could you please:

• move the flag to be --experimental- (this is mainly because right now scalibr only supports skipping directories, not files which'll hopefully be able to change but in the meantime this will mean we can do breaking changes if we go another direction instead)
• update the docs to reflect that this only applies to directories, not files
• add a couple of cmd tests
• update the syntax to use :, matching our --lockfile flag, which'll also let us use DirsToSkip too
  • if the string is prefixed with : or nothing, it should be applied as a directory (DirsToSkip)
  • if the string is prefixed with g:, it should be applied as a glob (SkipDirGlob)
  • if the string is prefixed with r:, it should be applied as a regex (SkipDirRegex)

[another-rex]

if the string is prefixed with :, it should be applied as a directory (DirsToSkip)

to clarify, also if it is not prefixed, use DirsToSkip, the prefixing with : is just an escape hatch incase your file path itself has a : inside.

[Ankitsinghsisodya]

@another-rex sir, I have done the changes suggested you.

[Ankitsinghsisodya]

I initially removed the type alias as you suggested, but CI failed with a depguard error:

import 'regexp' is not allowed from list 'regexp':
Use github.com/google/osv-scanner/v2/internal/cachedregexp instead (depguard)

The exclude.go file needs the *regexp.Regexp type for the struct field, but can't import regexp directly due to the lint rule. Adding the type alias back in cachedregexp allows other packages to use *cachedregexp.Regexp without violating depguard.

Should I look into adding an exception to the depguard config for this file, or is the type alias acceptable given this constraint?

[G-Rath]

right yeah of course - I think for now let's go with adding the exception as that mirrors what we're doing elsewhere for this situation, though I think I'll revisit that after this as it'll probably be better to just re-export stuff from cachedregexp and/or try to use the forbidigo linter instead

(I had forgotten we pretty much don't use regexp outside of cachedregexp.MustCompile 😅)

[G-Rath]

looking good, we just also do want to change the cmd flag to use exclude too

[another-rex]

Looks good!

[another-rex]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new experimental flag --experimental-exclude to allow excluding paths from scanning. The implementation is well-structured, with new logic for parsing exclusion patterns, comprehensive tests, and clear documentation. I've found one issue with how absolute paths are handled, which could lead to silent failures. My detailed feedback is in the review comments. Also, note that the PR description and title mention --exclude, but the implementation uses --experimental-exclude, which is correctly reflected in the documentation.

[another-rex]

Hmm the requested interactions not found errors are a bit odd in the CI, because it looks like you have added the new interactions to the cassettes. @G-Rath Can you take a look at why that could be happening?

[G-Rath]

I think they might just need to be regenerated, as they're still using the old X-Test-Name and it doesn't look like there are interactions for the exclude_with_multiple_exact_directories and exclude_with_multiple_pattern_types tests.

First I'd do is just revert all changes to cmd/osv-scanner/scan/source/testdata/cassettes/TestCommand.yaml, re-run the suite, and commit the results

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.76923% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.76%. Comparing base (f9a43d6) to head (11257de).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/osvscanner/exclude.go	91.66%	2 Missing and 2 partials ⚠️
internal/cachedregexp/regex.go	77.77%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2458      +/-   ##
==========================================
+ Coverage   67.72%   67.76%   +0.03%     
==========================================
  Files         172      173       +1     
  Lines       13343    13414      +71     
==========================================
+ Hits         9037     9090      +53     
- Misses       3592     3604      +12     
- Partials      714      720       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[another-rex]

Apologies for the delay, tests are mostly passing now, though there does seem to be a legitimate issue on Windows.

Is it easy for you to test on Windows to debug this? Otherwise we can look into it when we have time available.

[Ankitsinghsisodya]

@another-rex Sorry to say but I can only test on macos and Linux(ubuntu).

[another-rex]

@G-Rath Can you take a look at the windows error and see if it can be resolved easily?

[G-Rath]

@another-rex this is why Windows is unhappy, because r:/ and g:/ are absolute paths on Windows

Solutions that come to mind are:

1. we assume a lowercase r and g are our prefixes
   • Windows drive letters are case-insensitive and usually represented in uppercase
2. we have a double colon rule (i.e. have both r::/ and r:/ be valid)
   • yuck, and yet more of a bandaid
3. we make it a requirement to prefix with : if you're doing absolute paths, at least on Windows
   • this'd be inconsistent with our logic elsewhere, though we could change that too..
4. we change the symbol
   • : feels like the most intuitive symbol to me though for us to be using 😞
5. we explicitly check for r: and g:, and say Windows users have to prefix with : if they want to pass in those driver letters in lowercase
6. we do nothing (at least for now), and tell people their regexps need to start with something e.g. .*

Those last two seem like the most straightforward for the time being (given this is experimental and all)

[another-rex]

I think 5 is probably the best way to go here, and on windows just force uppercase to be used for drive names if they want to have drive r: or drive g: to be interpreted as drives.

Outdated