feat: support auto wildcard detection across multiple domains#960
Open
supreme2580 wants to merge 4 commits intoprojectdiscovery:devfrom
Open
feat: support auto wildcard detection across multiple domains#960supreme2580 wants to merge 4 commits intoprojectdiscovery:devfrom
supreme2580 wants to merge 4 commits intoprojectdiscovery:devfrom
Conversation
Introduce auto-wildcard flag to enable automatic wildcard detection based on domain extraction. Modify extractDomain function to handle both URLs and plain domains, extracting base domain (e.g., sub.example.com -> example.com). This eliminates the need to manually specify wildcard domain when processing mixed input formats.
WalkthroughAdds an optional --auto-wildcard flag and wiring to derive wildcard domains from hosts at runtime; enhances extractDomain to return eTLD+1 with fallbacks; updates wildcard detection/handling across runner/worker; adds tests; updates go.mod and .gitignore. Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as CLI (flag)
participant Runner as Runner
participant Extractor as DomainExtractor
participant Worker as DNSWorker
participant Store as ResultStore
CLI->>Runner: start with --auto-wildcard
Runner->>Runner: set AutoWildcard = true
Runner->>Worker: dispatch lookup request (host)
alt AutoWildcard enabled
Worker->>Extractor: extractDomain(host)
Extractor-->>Worker: wildcardDomain (eTLD+1)
else AutoWildcard disabled
Worker-->>Worker: use configured WildcardDomain
end
Worker->>Worker: perform DNS query
Worker-->>Runner: return answers + wildcardDomain
Runner->>Runner: filter results by wildcardDomain
Runner->>Store: persist non-wildcard results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
internal/runner/options.go (1)
309-311: Consider adding stream mode validation forAutoWildcard.
WildcardDomainis explicitly blocked in stream mode (line 309-310), butAutoWildcardisn't checked. Since wildcard filtering requires post-scan processing thatrunStream()doesn't perform, enabling--auto-wildcardwith--streamwould silently have no effect.💡 Proposed fix
if options.WildcardDomain != "" { gologger.Fatal().Msgf("wildcard not supported in stream mode") } + if options.AutoWildcard { + gologger.Fatal().Msgf("auto-wildcard not supported in stream mode") + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@internal/runner/options.go` around lines 309 - 311, The stream mode validation currently blocks options.WildcardDomain but misses options.AutoWildcard; update the validation in the same place where options.WildcardDomain is checked to also detect options.AutoWildcard and abort with the same fatal-style message (e.g., call gologger.Fatal().Msgf("wildcard not supported in stream mode")) so that enabling --auto-wildcard with runStream() fails loudly; reference the options.AutoWildcard field and the runStream() context when adding this check.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@internal/runner/util.go`:
- Around line 46-65: The current last-two-labels heuristic in the
domain-extraction logic (used in functions around util.go that parse
URL/hostname before wildcard trimming) fails for multi-part TLDs and
trailing-dot hostnames; replace that logic to use
golang.org/x/net/publicsuffix.EffectiveTLDPlusOne: first normalize the input by
trimming any trailing dot(s), if the input is not a full URL keep treating it as
a hostname, parse URL when needed to get u.Hostname(), then call
publicsuffix.EffectiveTLDPlusOne(hostname) and return that result; if
EffectiveTLDPlusOne returns an error or empty string, fall back to a safe join
of the last two labels of the cleaned hostname as before; add the import for
golang.org/x/net/publicsuffix and ensure all calls reference the same helper
function (e.g., your domain-extraction function) so downstream code like
TrimSuffix in wildcard.go and comparisons in runner.go use the corrected value.
---
Nitpick comments:
In `@internal/runner/options.go`:
- Around line 309-311: The stream mode validation currently blocks
options.WildcardDomain but misses options.AutoWildcard; update the validation in
the same place where options.WildcardDomain is checked to also detect
options.AutoWildcard and abort with the same fatal-style message (e.g., call
gologger.Fatal().Msgf("wildcard not supported in stream mode")) so that enabling
--auto-wildcard with runStream() fails loudly; reference the
options.AutoWildcard field and the runStream() context when adding this check.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7f9571f0-13d3-4d49-8b19-1027c637f1d7
📒 Files selected for processing (5)
.gitignoreinternal/runner/options.gointernal/runner/runner.gointernal/runner/util.gointernal/runner/wildcard.go
Replace manual domain parsing with golang.org/x/net/publicsuffix.EffectiveTLDPlusOne for accurate eTLD+1 extraction. This correctly handles complex TLDs like .co.uk and .com.au. Add comprehensive test cases to validate the behavior. Update Go version to 1.25.0 and bump dependency versions accordingly.
Downgrade Go version from 1.25.0 to 1.24.0 with toolchain 1.24.5 to maintain compatibility with the ecosystem. Update multiple projectdiscovery dependencies to older versions and adjust transitive dependencies accordingly. Add missing golang.org/x/net and google.golang.org/protobuf dependencies that are required by the downgraded dependency graph.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@go.mod`:
- Line 125: Update the golang.org/x/oauth2 module from v0.11.0 to v0.27.0:
change the require entry for golang.org/x/oauth2 to v0.27.0 (or run `go get
golang.org/x/oauth2@v0.27.0`) and then regenerate the module graph by running
`go mod tidy` (or `go get` followed by `go mod tidy`) to update go.sum and
dependencies; ensure CI builds to verify no breakages where oauth2 is used.
- Line 122: Update the golang.org/x/crypto module reference from v0.31.0 to
v0.45.0 (or later) in go.mod to mitigate the listed CVEs; run `go get
golang.org/x/crypto@v0.45.0` (or `go get -u golang.org/x/crypto`), then run `go
mod tidy` to update go.sum, and run the test suite/build to verify nothing
breaks (look for the dependency entry "golang.org/x/crypto" in go.mod to
change).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 6a517248-cf68-4df1-ae68-befb38cef363
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (1)
go.mod
Update several golang.org/x/* modules to newer versions: - golang.org/x/net from v0.33.0 to v0.47.0 - golang.org/x/crypto from v0.31.0 to v0.45.0 - golang.org/x/sys from v0.28.0 to v0.38.0 - golang.org/x/text from v0.21.0 to v0.31.0 - golang.org/x/mod from v0.18.0 to v0.29.0 - golang.org/x/tools from v0.22.0 to v0.38.0 - golang.org/x/sync from v0.10.0 to v0.18.0 - golang.org/x/term from v0.27.0 to v0.37.0 - golang.org/x/oauth2 from v0.11.0 to v0.27.0 Remove unused github.com/golang/protobuf, google.golang.org/appengine, and google.golang.org/protobuf dependencies to keep the dependency tree clean.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@go.mod`:
- Line 111: The go.mod downgrade of github.com/weppos/publicsuffix-go to v0.30.1
is inconsistent with the new extractDomain change that relies on publicsuffix
logic; update the go.mod to a current compatible version (e.g. v0.40.x) or
remove the explicit older indirect requirement so the module graph resolves the
newer publicsuffix implementation, then run go mod tidy to refresh sums; ensure
extractDomain references the golang.org/x/net/publicsuffix behavior you expect
after the dependency change.
- Line 20: Review the commit or dependency changes that replaced
github.com/projectdiscovery/retryabledns v1.0.111 with v1.0.102: check the
go.sum/history/replace directives, module upgrades in nearby commits, and
CI/build logs for errors; verify differences between v1.0.111 and v1.0.102 by
inspecting the upstream changelog/commit history for retryabledns and confirm
any behavioral/regression reasons for pinning, focusing on the Options struct
and NewWithOptions() API, and then either restore the newer version (v1.0.111+)
if safe or add a concise comment/PR note explaining the explicit downgrade and
the rationale (including any observed bug/regression) so future reviewers
understand why v1.0.102 was chosen.
- Around line 12-24: The go.mod changes show unintended downgrades of several
projectdiscovery/* modules; verify whether these version changes are intentional
and either pin the intended versions or document the rationale: inspect and
update go.mod to the correct versions for
github.com/projectdiscovery/{cdncheck,gologger,hmap,mapcidr,ratelimit,retryabledns,utils}
(or add a clear commit message/CHANGELOG entry explaining the downgrade), run
`go get`/`go mod tidy` to lock them, then run full tests and static checks to
confirm API compatibility with code that imports utils subpackages (fileutil,
iputil, mapsutil, sliceutil) and with retryabledns.CheckInternalIPs and the
retryabledns.DNSData unmarshalling; if incompatibilities are found, either
restore newer versions or adapt call sites accordingly and document the
decision.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 772728c2-6fa5-445c-8cf7-3b47ee17790b
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (1)
go.mod
Comment on lines
+12
to
+24
| github.com/projectdiscovery/cdncheck v1.1.36 | ||
| github.com/projectdiscovery/clistats v0.1.1 | ||
| github.com/projectdiscovery/goconfig v0.0.1 | ||
| github.com/projectdiscovery/goflags v0.1.74 | ||
| github.com/projectdiscovery/gologger v1.1.62 | ||
| github.com/projectdiscovery/hmap v0.0.98 | ||
| github.com/projectdiscovery/mapcidr v1.1.97 | ||
| github.com/projectdiscovery/ratelimit v0.0.82 | ||
| github.com/projectdiscovery/retryabledns v1.0.111 | ||
| github.com/projectdiscovery/utils v0.7.3 | ||
| github.com/projectdiscovery/gologger v1.1.56 | ||
| github.com/projectdiscovery/hmap v0.0.90 | ||
| github.com/projectdiscovery/mapcidr v1.1.34 | ||
| github.com/projectdiscovery/ratelimit v0.0.81 | ||
| github.com/projectdiscovery/retryabledns v1.0.102 | ||
| github.com/projectdiscovery/utils v0.4.20 | ||
| github.com/rs/xid v1.5.0 | ||
| github.com/stretchr/testify v1.11.1 | ||
| golang.org/x/net v0.47.0 |
There was a problem hiding this comment.
❓ Verification inconclusive
Script executed:
#!/bin/bash
# Check git history for these dependency versions to determine if this is a revert
echo "=== Checking cdncheck version history ==="
git log --all --oneline -S 'github.com/projectdiscovery/cdncheck v1.1.36' -- go.mod | head -5
echo ""
echo "=== Checking utils version history ==="
git log --all --oneline -S 'github.com/projectdiscovery/utils v0.4.20' -- go.mod | head -5
echo ""
echo "=== Checking retryabledns version history ==="
git log --all --oneline -S 'github.com/projectdiscovery/retryabledns v1.0.102' -- go.mod | head -5
echo ""
echo "=== Checking recent go.mod changes on dev branch ==="
git log --oneline origin/dev -- go.mod | head -10Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Script execution failed with error: Request timed out after 120000ms (requestId=bf2c281a-5658-4f18-8353-f4f4ae26df0e)
Script executed:
# First, let's read the go.mod file to confirm the versions
head -30 go.modRepository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
module github.com/projectdiscovery/dnsx
go 1.24.0
toolchain go1.24.5
require (
github.com/logrusorgru/aurora v2.0.3+incompatible
github.com/miekg/dns v1.1.62
github.com/pkg/errors v0.9.1
github.com/projectdiscovery/asnmap v1.1.1
github.com/projectdiscovery/cdncheck v1.1.36
github.com/projectdiscovery/clistats v0.1.1
github.com/projectdiscovery/goconfig v0.0.1
github.com/projectdiscovery/goflags v0.1.74
github.com/projectdiscovery/gologger v1.1.56
github.com/projectdiscovery/hmap v0.0.90
github.com/projectdiscovery/mapcidr v1.1.34
github.com/projectdiscovery/ratelimit v0.0.81
github.com/projectdiscovery/retryabledns v1.0.102
github.com/projectdiscovery/utils v0.4.20
github.com/rs/xid v1.5.0
github.com/stretchr/testify v1.11.1
golang.org/x/net v0.47.0
)
require (
aead.dev/minisign v0.2.0 // indirect
github.com/Masterminds/semver/v3 v3.2.1 // indirect
github.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057 // indirect
Script executed:
# Check the full go.mod file to see all dependencies
wc -l go.modRepository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
134 go.mod
Script executed:
# Search for usage of the key downgraded libraries in the codebase
rg "github.com/projectdiscovery/utils" --type go -A 2 | head -30Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
internal/runner/banner.go: "github.com/projectdiscovery/utils/auth/pdcp"
internal/runner/banner.go: updateutils "github.com/projectdiscovery/utils/update"
internal/runner/banner.go-)
internal/runner/banner.go-
--
internal/runner/util.go: fileutil "github.com/projectdiscovery/utils/file"
internal/runner/util.go- "golang.org/x/net/publicsuffix"
internal/runner/util.go-)
--
internal/runner/runner_test.go: stringsutil "github.com/projectdiscovery/utils/strings"
internal/runner/runner_test.go- "github.com/stretchr/testify/require"
internal/runner/runner_test.go-)
--
internal/runner/runner.go: fileutil "github.com/projectdiscovery/utils/file"
internal/runner/runner.go: iputil "github.com/projectdiscovery/utils/ip"
internal/runner/runner.go: mapsutil "github.com/projectdiscovery/utils/maps"
internal/runner/runner.go: sliceutil "github.com/projectdiscovery/utils/slice"
internal/runner/runner.go-)
internal/runner/runner.go-
--
internal/runner/options.go: "github.com/projectdiscovery/utils/auth/pdcp"
internal/runner/options.go: "github.com/projectdiscovery/utils/env"
internal/runner/options.go: fileutil "github.com/projectdiscovery/utils/file"
internal/runner/options.go: updateutils "github.com/projectdiscovery/utils/update"
internal/runner/options.go-)
internal/runner/options.go-
--
internal/runner/healthcheck.go: fileutil "github.com/projectdiscovery/utils/file"
internal/runner/healthcheck.go-)
internal/runner/healthcheck.go-
Script executed:
# Check for usage of retryabledns
rg "retryabledns" --type go -B 2 -A 2 | head -30Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
internal/runner/options.go- options := &Options{}
internal/runner/options.go- flagSet := goflags.NewFlagSet()
internal/runner/options.go: flagSet.SetDescription(`dnsx is a fast and multi-purpose DNS toolkit allow to run multiple probes using retryabledns library.`)
internal/runner/options.go-
internal/runner/options.go- flagSet.CreateGroup("input", "Input",
--
internal/runner/options.go- }
internal/runner/options.go- options.Response = true
internal/runner/options.go: // the ANY query type is not supported by the retryabledns library,
internal/runner/options.go- // thus it's hard to filter the results when it's used in combination with other query types
internal/runner/options.go- options.ExcludeType = append(options.ExcludeType, "any")
--
internal/runner/runner.go- "github.com/projectdiscovery/mapcidr/asn"
internal/runner/runner.go- "github.com/projectdiscovery/ratelimit"
internal/runner/runner.go: "github.com/projectdiscovery/retryabledns"
internal/runner/runner.go- fileutil "github.com/projectdiscovery/utils/file"
internal/runner/runner.go- iputil "github.com/projectdiscovery/utils/ip"
--
internal/runner/runner.go-
internal/runner/runner.go-func New(options *Options) (*Runner, error) {
internal/runner/runner.go: retryabledns.CheckInternalIPs = true
internal/runner/runner.go-
internal/runner/runner.go- dnsxOptions := dnsx.DefaultOptions
--
internal/runner/runner.go- // prepare in memory structure similarly to shuffledns
internal/runner/runner.go- r.hm.Scan(func(k, v []byte) error {
internal/runner/runner.go: var dnsdata retryabledns.DNSData
internal/runner/runner.go- if err := json.Unmarshal(v, &dnsdata); err != nil {
internal/runner/runner.go- // the item has no record - ignore
--
Script executed:
# Look for any PR description or comments explaining the downgrades
find . -name "*.md" -type f -exec grep -l "downgrade\|version\|dependency" {} \; | head -10Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
./.github/ISSUE_TEMPLATE/issue-report.md
./README.md
Script executed:
# Check the current branch and recent commits
git status
git log -5 --onelineRepository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
HEAD detached at FETCH_HEAD
nothing to commit, working tree clean
9ba00e5 build: update Go dependencies to newer versions
Script executed:
# Check if there's a PR template or description
find . -name "*.md" -type f | xargs grep -l "auto wildcard\|publicsuffix" 2>/dev/null | head -5Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
Script executed:
# Check the actual PR or branch information
git log --all --oneline --graph | head -20Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
* 9ba00e5 build: update Go dependencies to newer versions
* fe80b18 version update
* b05668c Merge pull request `#916` from projectdiscovery/fix/omit-raw-flag
|\
| * 00c85d3 fix: remove RawResp from JSON output completely
* | 62d3a7a chore(deps): bump the modules group with 8 updates
* | 31932fd chore(deps): bump the modules group with 8 updates (`#922`)
* | 3ba9878 chore(deps): bump the modules group across 1 directory with 8 updates
* | 8c02098 chore(deps): bump the modules group with 4 updates (`#915`)
|/
* 6aaab05 chore(deps): bump the modules group with 6 updates
* d2f53f4 chore(deps): bump the modules group across 1 directory with 9 updates
* 037e267 Merge pull request `#903` from jjhwan-h/fix/wd
|\
| * 44a9b20 Merge branch 'dev' into pr/903
| |\
| |/
|/|
* | 31d97f2 Merge pull request `#907` from geeknik/dev
|\ \
Script executed:
# Look for any comments in go.mod about version choices
cat go.mod | grep -A 2 -B 2 "utils\|retryabledns\|cdncheck"Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
github.com/pkg/errors v0.9.1
github.com/projectdiscovery/asnmap v1.1.1
github.com/projectdiscovery/cdncheck v1.1.36
github.com/projectdiscovery/clistats v0.1.1
github.com/projectdiscovery/goconfig v0.0.1
--
github.com/projectdiscovery/mapcidr v1.1.34
github.com/projectdiscovery/ratelimit v0.0.81
github.com/projectdiscovery/retryabledns v1.0.102
github.com/projectdiscovery/utils v0.4.20
github.com/rs/xid v1.5.0
github.com/stretchr/testify v1.11.1
Script executed:
# Check if there's a go.sum that might give us hints about why these versions were chosen
grep "github.com/projectdiscovery/utils" go.sum | head -10Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
github.com/projectdiscovery/utils v0.4.20 h1:7Fmjb+4YZJSzn7bL21sjF3wAR53eSi7VdAfDkDBUUwY=
github.com/projectdiscovery/utils v0.4.20/go.mod h1:RnC23+hI8j4drZFHQpMX92hV9++9d/yBeNr1pzcbF7Y=
Script executed:
# Show the previous version of go.mod to see what was before
git show HEAD~1:go.mod | grep -E "utils|retryabledns|cdncheck" | head -10Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stderr:
fatal: invalid object name 'HEAD~1'.
Script executed:
# Check git blame or history for when these were last downgraded
git log --all --oneline --follow -- go.mod | head -20Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
9ba00e5 build: update Go dependencies to newer versions
62d3a7a chore(deps): bump the modules group with 8 updates
31932fd chore(deps): bump the modules group with 8 updates (`#922`)
3ba9878 chore(deps): bump the modules group across 1 directory with 8 updates
8c02098 chore(deps): bump the modules group with 4 updates (`#915`)
6aaab05 chore(deps): bump the modules group with 6 updates
d2f53f4 chore(deps): bump the modules group across 1 directory with 9 updates
da2aebb chore(deps): bump the modules group across 1 directory with 6 updates
92dbe0b bump
ed433b2 chore(deps): bump the modules group across 1 directory with 8 updates
16bbeb8 bump retryabledns
cfdced5 chore(deps): bump the modules group with 9 updates
a596d0e bump go
ea44ab2 chore(deps): bump the modules group across 1 directory with 10 updates
f3d64a4 chore(deps): bump github.com/projectdiscovery/gologger
54cae21 chore(deps): bump github.com/projectdiscovery/retryablehttp-go
48bb5be chore(deps): bump github.com/projectdiscovery/fastdialer
b3309b6 chore(deps): bump github.com/projectdiscovery/hmap from 0.0.75 to 0.0.77
06e3a34 chore(deps): bump github.com/projectdiscovery/retryabledns
870631b chore(deps): bump github.com/projectdiscovery/fastdialer
Script executed:
# Show detailed diff of the go.mod changes in this commit
git show HEAD -- go.mod | head -80Repository: projectdiscovery/dnsx
Repository: projectdiscovery/dnsx
Exit code: 0
stdout:
commit 9ba00e52ebc1f20d5ed1bcb683bd93767a8ebcc8
Author: supreme2580 <victoromorogbe69@gmail.com>
Date: Fri Mar 13 12:07:14 2026 +0100
build: update Go dependencies to newer versions
Update several golang.org/x/* modules to newer versions:
- golang.org/x/net from v0.33.0 to v0.47.0
- golang.org/x/crypto from v0.31.0 to v0.45.0
- golang.org/x/sys from v0.28.0 to v0.38.0
- golang.org/x/text from v0.21.0 to v0.31.0
- golang.org/x/mod from v0.18.0 to v0.29.0
- golang.org/x/tools from v0.22.0 to v0.38.0
- golang.org/x/sync from v0.10.0 to v0.18.0
- golang.org/x/term from v0.27.0 to v0.37.0
- golang.org/x/oauth2 from v0.11.0 to v0.27.0
Remove unused github.com/golang/protobuf, google.golang.org/appengine, and google.golang.org/protobuf dependencies to keep the dependency tree clean.
diff --git a/go.mod b/go.mod
new file mode 100644
index 0000000..a3b6d60
--- /dev/null
+++ b/go.mod
@@ -0,0 +1,134 @@
+module github.com/projectdiscovery/dnsx
+
+go 1.24.0
+
+toolchain go1.24.5
+
+require (
+ github.com/logrusorgru/aurora v2.0.3+incompatible
+ github.com/miekg/dns v1.1.62
+ github.com/pkg/errors v0.9.1
+ github.com/projectdiscovery/asnmap v1.1.1
+ github.com/projectdiscovery/cdncheck v1.1.36
+ github.com/projectdiscovery/clistats v0.1.1
+ github.com/projectdiscovery/goconfig v0.0.1
+ github.com/projectdiscovery/goflags v0.1.74
+ github.com/projectdiscovery/gologger v1.1.56
+ github.com/projectdiscovery/hmap v0.0.90
+ github.com/projectdiscovery/mapcidr v1.1.34
+ github.com/projectdiscovery/ratelimit v0.0.81
+ github.com/projectdiscovery/retryabledns v1.0.102
+ github.com/projectdiscovery/utils v0.4.20
+ github.com/rs/xid v1.5.0
+ github.com/stretchr/testify v1.11.1
+ golang.org/x/net v0.47.0
+)
+
+require (
+ aead.dev/minisign v0.2.0 // indirect
+ github.com/Masterminds/semver/v3 v3.2.1 // indirect
+ github.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057 // indirect
+ github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 // indirect
+ github.com/STARRY-S/zip v0.2.1 // indirect
+ github.com/VividCortex/ewma v1.2.0 // indirect
+ github.com/akrylysov/pogreb v0.10.1 // indirect
+ github.com/alecthomas/chroma/v2 v2.14.0 // indirect
+ github.com/andybalholm/brotli v1.1.1 // indirect
+ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
+ github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
+ github.com/aymerick/douceur v0.2.0 // indirect
+ github.com/bodgit/plumbing v1.3.0 // indirect
+ github.com/bodgit/sevenzip v1.6.0 // indirect
+ github.com/bodgit/windows v1.0.1 // indirect
+ github.com/charmbracelet/glamour v0.8.0 // indirect
+ github.com/charmbracelet/lipgloss v0.13.0 // indirect
+ github.com/charmbracelet/x/ansi v0.3.2 // indirect
+ github.com/cheggaaa/pb/v3 v3.1.4 // indirect
+ github.com/cloudflare/circl v1.6.1 // indirect
+ github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
+ github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+ github.com/dimchansky/utfbom v1.1.1 // indirect
+ github.com/dlclark/regexp2 v1.11.5 // indirect
+ github.com/docker/go-units v0.5.0 // indirect
+ github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect
+ github.com/fatih/color v1.15.0 // indirect
+ github.com/gaissmai/bart v0.26.0 // indirect
Undocumented dependency downgrades: Verify these versions are intentional and compatible.
This commit updates golang.org/x modules to newer versions but simultaneously downgrades 7 projectdiscovery/* packages without documentation:
cdncheck: v1.2.13 → v1.1.36gologger: v1.1.62 → v1.1.56hmap: v0.0.98 → v0.0.90mapcidr: v1.1.97 → v1.1.34ratelimit: v0.0.82 → v0.0.81retryabledns: v1.0.111 → v1.0.102utils: v0.7.3 → v0.4.20
The commit message only mentions golang.org/x updates and does not explain these downgrades. They appear to be unintentional side effects from Go's dependency resolver. Confirm that these downgraded versions are:
- Intentional, with documented rationale
- API-compatible with code that imports utils subpackages (fileutil, iputil, mapsutil, sliceutil), calls retryabledns.CheckInternalIPs, and unmarshals retryabledns.DNSData
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@go.mod` around lines 12 - 24, The go.mod changes show unintended downgrades
of several projectdiscovery/* modules; verify whether these version changes are
intentional and either pin the intended versions or document the rationale:
inspect and update go.mod to the correct versions for
github.com/projectdiscovery/{cdncheck,gologger,hmap,mapcidr,ratelimit,retryabledns,utils}
(or add a clear commit message/CHANGELOG entry explaining the downgrade), run
`go get`/`go mod tidy` to lock them, then run full tests and static checks to
confirm API compatibility with code that imports utils subpackages (fileutil,
iputil, mapsutil, sliceutil) and with retryabledns.CheckInternalIPs and the
retryabledns.DNSData unmarshalling; if incompatibilities are found, either
restore newer versions or adapt call sites accordingly and document the
decision.
| github.com/projectdiscovery/hmap v0.0.90 | ||
| github.com/projectdiscovery/mapcidr v1.1.34 | ||
| github.com/projectdiscovery/ratelimit v0.0.81 | ||
| github.com/projectdiscovery/retryabledns v1.0.102 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check retryabledns Options struct and NewWithOptions in v1.0.102
echo "=== Cloning retryabledns v1.0.102 ==="
git clone --depth 1 --branch v1.0.102 https://github.com/projectdiscovery/retryabledns /tmp/retryabledns-v1.0.102 2>/dev/null
if [ -d /tmp/retryabledns-v1.0.102 ]; then
echo ""
echo "Checking Options struct fields:"
rg -A 20 'type Options struct' /tmp/retryabledns-v1.0.102
echo ""
echo "Checking NewWithOptions function signature:"
rg -B 2 -A 5 'func NewWithOptions' /tmp/retryabledns-v1.0.102
rm -rf /tmp/retryabledns-v1.0.102
fiRepository: projectdiscovery/dnsx
Length of output: 4247
Investigate why retryabledns was downgraded from v1.0.111 to v1.0.102.
The downgrade spans 9 patch versions, which could revert bug fixes. API compatibility has been confirmed—v1.0.102 includes all required Options struct fields (BaseResolvers, MaxRetries, Hostsfile, Proxy, Timeout) and the NewWithOptions() function signature. However, the downgrade itself is unusual and suggests either a regression in v1.0.103+ or a workaround. Clarify the reason for this change to avoid introducing unforeseen issues or technical debt.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@go.mod` at line 20, Review the commit or dependency changes that replaced
github.com/projectdiscovery/retryabledns v1.0.111 with v1.0.102: check the
go.sum/history/replace directives, module upgrades in nearby commits, and
CI/build logs for errors; verify differences between v1.0.111 and v1.0.102 by
inspecting the upstream changelog/commit history for retryabledns and confirm
any behavioral/regression reasons for pinning, focusing on the Options struct
and NewWithOptions() API, and then either restore the newer version (v1.0.111+)
if safe or add a concise comment/PR note explaining the explicit downgrade and
the rationale (including any observed bug/regression) so future reviewers
understand why v1.0.102 was chosen.
| github.com/ulikunitz/xz v0.5.15 // indirect | ||
| github.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39 // indirect | ||
| github.com/ulikunitz/xz v0.5.12 // indirect | ||
| github.com/weppos/publicsuffix-go v0.30.1 // indirect |
There was a problem hiding this comment.
Inconsistent: Downgrading publicsuffix-go while adding publicsuffix-based feature.
This PR enhances extractDomain to use eTLD+1 resolution via golang.org/x/net/publicsuffix (line 24), yet downgrades the indirect publicsuffix-go dependency from v0.40.x to v0.30.1. While publicsuffix-go is indirect, downgrading a domain-suffix library during a feature that relies on public suffix logic is inconsistent and warrants explanation.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@go.mod` at line 111, The go.mod downgrade of
github.com/weppos/publicsuffix-go to v0.30.1 is inconsistent with the new
extractDomain change that relies on publicsuffix logic; update the go.mod to a
current compatible version (e.g. v0.40.x) or remove the explicit older indirect
requirement so the module graph resolves the newer publicsuffix implementation,
then run go mod tidy to refresh sums; ensure extractDomain references the
golang.org/x/net/publicsuffix behavior you expect after the dependency change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR implements automatic wildcard detection and filtering, similar to the functionality in PureDNS. It allows
dnsxto automatically identify the base domain for each input host and perform wildcard probing/filtering across multiple domains in a single run.Key Changes
--auto-wildcard(-aw) to enable automatic base domain extraction and wildcard verification.extractDomainutility in util.go to accurately identify base domains (e.g.,a.b.example.com->example.com) for both URL and raw string inputs.Usage
This feature is fully optional and ensures that existing workflows using
-wdor standard resolution remain unaffected./claim #924
closes: #924
Summary by CodeRabbit
New Features
Improvements
Tests
Chores