feat: auto wildcard domain detection for multi-domain runs (#924)#949
Open
Jujubee-LLM wants to merge 6 commits intoprojectdiscovery:devfrom
Open
feat: auto wildcard domain detection for multi-domain runs (#924)#949Jujubee-LLM wants to merge 6 commits intoprojectdiscovery:devfrom
Jujubee-LLM wants to merge 6 commits intoprojectdiscovery:devfrom
Conversation
Neo - PR Security ReviewNo security issues found Highlights
Hardening Notes
Comment |
WalkthroughAdds an optional AutoWildcard mode and wiring to auto-detect and handle wildcard root domains per host. Introduces Changes
Sequence Diagram(s)sequenceDiagram
participant Runner as Runner
participant TaskQ as WildcardWorker
participant DNS as DNSResolver
participant Results as ResultProcessor
Runner->>Runner: enumerate hosts
Runner->>Runner: getWildcardDomainForHost(host) / normalizeHostname
Runner->>TaskQ: send wildcardTask(host, wildcardDomain)
TaskQ->>DNS: query host (A/other qtypes as configured)
DNS-->>TaskQ: response (records/wildcard indicators)
TaskQ-->>Runner: report wildcard detection for host
Runner->>Results: apply host->wildcardDomain map, filter/mark apex wildcards
Results-->>Runner: output filtered results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
internal/runner/runner.go (2)
517-524: Consider adding debug logging when hosts are skipped from wildcard processing.When
getWildcardDomainForHostreturnsfalse(e.g., for IP addresses or hosts with ports), those hosts silently bypass wildcard filtering and pass through in the output loop. While this is correct behavior, adding a debug log would aid troubleshooting when users encounter unexpected pass-through hosts.📝 Suggested logging
wildcardDomain, ok := r.getWildcardDomainForHost(host) if !ok { + gologger.Debug().Msgf("Skipping wildcard check for host (no valid domain): %s\n", host) continue }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@internal/runner/runner.go` around lines 517 - 524, Add a debug log when getWildcardDomainForHost(host) returns false so skipped hosts are visible; specifically, in the loop around getWildcardDomainForHost (the code that populates hostToWildcardDomain, seen, and sends wildcardTask on r.wildcardworkerchan), emit a debug-level message indicating the host was skipped and the reason (e.g., "no wildcard domain" or "IP/contains port") before continue so operators can trace why a host was not queued for wildcard processing.
35-38: Consider adding a doc comment for the new type.The
wildcardTaskstruct is introduced as an internal pipeline message. A brief comment explaining its purpose would improve readability, especially clarifying thatdomainis the derived wildcard root (eTLD+1 or explicit).📝 Suggested documentation
+// wildcardTask carries a host and its associated wildcard domain +// through the wildcard detection pipeline. type wildcardTask struct { host string domain string }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@internal/runner/runner.go` around lines 35 - 38, Add a brief doc comment immediately above the wildcardTask type declaration explaining that wildcardTask is an internal pipeline message carrying the originating host (host) and the derived wildcard root (domain — eTLD+1 or explicit), describing its purpose and when/why domain is derived; place the comment directly above the type declaration for wildcardTask to improve readability.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@internal/runner/runner.go`:
- Around line 517-524: Add a debug log when getWildcardDomainForHost(host)
returns false so skipped hosts are visible; specifically, in the loop around
getWildcardDomainForHost (the code that populates hostToWildcardDomain, seen,
and sends wildcardTask on r.wildcardworkerchan), emit a debug-level message
indicating the host was skipped and the reason (e.g., "no wildcard domain" or
"IP/contains port") before continue so operators can trace why a host was not
queued for wildcard processing.
- Around line 35-38: Add a brief doc comment immediately above the wildcardTask
type declaration explaining that wildcardTask is an internal pipeline message
carrying the originating host (host) and the derived wildcard root (domain —
eTLD+1 or explicit), describing its purpose and when/why domain is derived;
place the comment directly above the type declaration for wildcardTask to
improve readability.
Jujubee-LLM
changed the title
Jujubee-LLM
changed the title
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
internal/runner/runner.go (1)
129-132: Avoid duplicate A queries when auto-wildcard is enabled.Line 131 appends
dns.TypeAunconditionally. IfAis already selected (e.g.,-aor--recon), this duplicates the same query type and adds unnecessary DNS traffic.🔧 Proposed refactor
// If no option is specified or wildcard filter has been requested use query type A if len(questionTypes) == 0 || options.WildcardDomain != "" || options.AutoWildcard { options.A = true - questionTypes = append(questionTypes, dns.TypeA) + if !sliceutil.Contains(questionTypes, dns.TypeA) { + questionTypes = append(questionTypes, dns.TypeA) + } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@internal/runner/runner.go` around lines 129 - 132, The code unconditionally appends dns.TypeA which can duplicate A queries; modify the block around questionTypes and options (the variables questionTypes, options.A, options.WildcardDomain, options.AutoWildcard) so that after setting options.A = true you only append dns.TypeA if it is not already present in questionTypes (e.g., check for existence using a small loop or helper like containsType(questionTypes, dns.TypeA) before append). Ensure dns.TypeA is added exactly once to avoid duplicate DNS traffic.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@internal/runner/runner.go`:
- Around line 60-62: The hostname normalization is currently not lowercasing,
causing case-sensitive apex comparisons to fail; update normalizeHostname to
return strings.ToLower(strings.TrimSuffix(strings.TrimSpace(host), ".")) and
ensure any apex comparisons use normalizeHostname(...) for both sides (where
apex/host are compared) so DNS labels are compared case-insensitively.
- Around line 451-454: When falling back for wildcard filtering in the block
that checks r.options.WildcardDomain or r.options.AutoWildcard, disable stream
mode on the options before invoking r.run() so prepareInput() consumes stdin
once and startWorkers() picks the buffered worker path instead of
InputWorkerStream(); in other words, set r.options.Stream = false (or call the
buffered/run variant) prior to calling r.run() to avoid re-reading exhausted
stdin by InputWorkerStream(), referencing r.run(), prepareInput(),
startWorkers(), InputWorkerStream(), and r.options.Stream.
---
Nitpick comments:
In `@internal/runner/runner.go`:
- Around line 129-132: The code unconditionally appends dns.TypeA which can
duplicate A queries; modify the block around questionTypes and options (the
variables questionTypes, options.A, options.WildcardDomain,
options.AutoWildcard) so that after setting options.A = true you only append
dns.TypeA if it is not already present in questionTypes (e.g., check for
existence using a small loop or helper like containsType(questionTypes,
dns.TypeA) before append). Ensure dns.TypeA is added exactly once to avoid
duplicate DNS traffic.
Comment on lines
+60
to
+62
| func normalizeHostname(host string) string { | ||
| return strings.TrimSuffix(strings.TrimSpace(host), ".") | ||
| } |
There was a problem hiding this comment.
Normalize host/domain case before apex comparison.
Line 608 currently compares case-sensitively. DNS labels are case-insensitive, so Example.COM vs example.com should still match.
🔧 Proposed fix
func normalizeHostname(host string) string {
- return strings.TrimSuffix(strings.TrimSpace(host), ".")
+ return strings.TrimSuffix(strings.ToLower(strings.TrimSpace(host)), ".")
}Also applies to: 607-609
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@internal/runner/runner.go` around lines 60 - 62, The hostname normalization
is currently not lowercasing, causing case-sensitive apex comparisons to fail;
update normalizeHostname to return
strings.ToLower(strings.TrimSuffix(strings.TrimSpace(host), ".")) and ensure any
apex comparisons use normalizeHostname(...) for both sides (where apex/host are
compared) so DNS labels are compared case-insensitively.
Comment on lines
+451
to
+454
| if r.options.WildcardDomain != "" || r.options.AutoWildcard { | ||
| gologger.Warning().Msgf("Wildcard filtering enabled in stream mode: falling back to buffered execution") | ||
| return r.run() | ||
| } |
There was a problem hiding this comment.
Stream fallback currently breaks stdin-backed wildcard runs.
At Line 453, r.run() is called while options.Stream remains true. run() calls prepareInput() (which consumes stdin), but startWorkers() then still picks InputWorkerStream(), which re-reads exhausted stdin. Result: zero processed tasks in stdin stream+wildcard mode.
🔧 Proposed fix
func (r *Runner) Run() error {
if r.options.Stream {
if r.options.WildcardDomain != "" || r.options.AutoWildcard {
gologger.Warning().Msgf("Wildcard filtering enabled in stream mode: falling back to buffered execution")
+ r.options.Stream = false
return r.run()
}
return r.runStream()
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if r.options.WildcardDomain != "" || r.options.AutoWildcard { | |
| gologger.Warning().Msgf("Wildcard filtering enabled in stream mode: falling back to buffered execution") | |
| return r.run() | |
| } | |
| func (r *Runner) Run() error { | |
| if r.options.Stream { | |
| if r.options.WildcardDomain != "" || r.options.AutoWildcard { | |
| gologger.Warning().Msgf("Wildcard filtering enabled in stream mode: falling back to buffered execution") | |
| r.options.Stream = false | |
| return r.run() | |
| } | |
| return r.runStream() | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@internal/runner/runner.go` around lines 451 - 454, When falling back for
wildcard filtering in the block that checks r.options.WildcardDomain or
r.options.AutoWildcard, disable stream mode on the options before invoking
r.run() so prepareInput() consumes stdin once and startWorkers() picks the
buffered worker path instead of InputWorkerStream(); in other words, set
r.options.Stream = false (or call the buffered/run variant) prior to calling
r.run() to avoid re-reading exhausted stdin by InputWorkerStream(), referencing
r.run(), prepareInput(), startWorkers(), InputWorkerStream(), and
r.options.Stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional per-host wildcard root detection for multi-domain input.
Previously, wildcard checks relied on a single configured
--wildcard-domain, which is not ideal when inputs span multiple root domains.With
--auto-wildcard, dnsx derives wildcard root domain per host (eTLD+1) and applies wildcard filtering against that resolved domain.Closes #924
Key changes
--auto-wildcard(Options.AutoWildcard)--wildcard-domainwhen providedpublicsuffix.EffectiveTLDPlusOnewhen--auto-wildcardis enabledwildcardTask)IsWildcard(host)->IsWildcard(host, wildcardDomain)--streamby falling back to buffered execution path when wildcard flags are enabledTests
Added/updated tests for:
Notes
No output format changes. Existing non-wildcard stream behavior remains unchanged.