Skip to content

Fix #4: 403 Permission Denied#5

Open
yunginnanet wants to merge 1 commit into
daehee:masterfrom
yunginnanet:fix-403
Open

Fix #4: 403 Permission Denied#5
yunginnanet wants to merge 1 commit into
daehee:masterfrom
yunginnanet:fix-403

Conversation

@yunginnanet

Copy link
Copy Markdown

Summary
This PR adds default HTTP headers (including a User-Agent and other relevant headers) to address the Forbidden error when scraping DoD web directories.

In essence this just reproduces the headers that my browser uses.

Changes

  1. Define a constant defUserAgent to mimic a modern browser.
  2. Initialize an http.Header object with several header fields:
    • User-Agent
    • Accept
    • Accept-Language
    • Upgrade-Insecure-Requests
    • Sec-Fetch-Site
    • Sec-Fetch-User
    • Accept-Encoding
    • Priority
    • Te

Potential Future Improvements
Add a list of UserAgent values and a bit of entropy to the header values in general. This should only be necessary if #4 gets reopened due to changes on DoD's end.

@yunginnanet yunginnanet changed the title Chore: deps Fix #4: 403 Permission Denied Apr 10, 2025
@Jah-yee

Jah-yee commented Jul 1, 2026

Copy link
Copy Markdown

Closing as duplicate of PR #6 which addresses a different issue (go install deprecation fix). Thanks for your contribution!

@Jah-yee

Jah-yee commented Jul 1, 2026

Copy link
Copy Markdown

Closing duplicate of PR#6 which is a cleaner, more targeted fix.

@Jah-yee

Jah-yee commented Jul 1, 2026

Copy link
Copy Markdown

Closing as duplicate. The go get→go install deprecation fix is better handled by PR #6 which specifically addresses the README deprecation.

@Jah-yee

Jah-yee commented Jul 1, 2026

Copy link
Copy Markdown

This PR (#5) fixes a 403 permission issue, but PR #6 specifically addresses the go get→go install deprecation in the README. Consider closing #5 in favor of #6 which addresses a different concern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants