Skip to content

Record ETag and Last-Modified for each URL #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
DilumAluthge wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Record ETag and Last-Modified for each URL #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5df1553
Record `ETag` and `Last-Modified` for each URL
DilumAluthge May 1, 2026
4d5f3c7
`Project.toml`: Add `URIs` as a direct dependency
DilumAluthge May 1, 2026
2d97116
`Manifest.toml`: re-resolve the manifest
DilumAluthge May 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Manifest.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

julia_version = "1.10.11"
manifest_format = "2.0"
project_hash = "297ab27881cec0a85c719648b28381580e85162c"
project_hash = "1faf98ec23df19c842aa27dbea894b9ec5b8c027"

[[deps.AbstractFFTs]]
deps = ["LinearAlgebra"]
Expand Down
1 change: 1 addition & 0 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce"
Tar = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
TimeZones = "f269a46b-ccf7-5d73-abea-4c690281aa53"
URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
WebCacheUtilities = "0c1c26de-fc5f-47ff-87a8-a157289a9bac"

[compat]
Expand Down
35 changes: 35 additions & 0 deletions src/VersionsJSONUtil.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ using HTTP, JSON, Pkg.BinaryPlatforms, WebCacheUtilities, SHA, Lazy
using Tar: Tar
import Pkg.BinaryPlatforms: triplet, arch
import Pkg.PlatformEngines: exe7z
using URIs: URIs, URI

"Wrapper types to define three jlext methods for portable, tarball and installer Windows"
struct WindowsPortable
Expand Down Expand Up @@ -143,6 +144,31 @@ function get_tags()
JSON.parse(String(read(tags_json_path)))
end

##### --------------------------------------------------------------------------------------
##### Get ETag and Last-Modified, so we know if we need to re-download and re-checksum files

Base.@kwdef struct HeadInfo
url::URI
etag::Union{String, Nothing}
last_modified::Union{String, Nothing}
end

function HeadInfo(url)
local response = nothing
try
response = HTTP.head(url)
catch
error("Encountered error when making HEAD request to URL: $url")
end
Comment on lines +157 to +162
Copy link
Copy Markdown
Member

@ararslan ararslan Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the idea behind intercepting an exception thrown by HTTP.head is to be able to ensure the URL is logged—is that right? As written, you lose what the actual error was. Instead, you could do something like this:

Suggested change
local response = nothing
try
response = HTTP.head(url)
catch
error("Encountered error when making HEAD request to URL: $url")
end
response = try
HTTP.head(url)
catch
@error "Encounted error when making HEAD request to URL: $url"
rethrow()
end

That said, errors from HTTP.jl generally do tell you what the URL was, so you could alternatively just do

Suggested change
local response = nothing
try
response = HTTP.head(url)
catch
error("Encountered error when making HEAD request to URL: $url")
end
response = HTTP.head(url)

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the original error still appears in the stacktrace, right? It'll be something like [our error] "caused by" [original error].

Copy link
Copy Markdown
Member

@ararslan ararslan Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh does it? I remember at some point the output from some errors doubled in length but it was never clear to me why or what makes it do that.

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me double-check locally to make sure.

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge May 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I tested on Julia 1.10, and the original error is still shown.

julia> try
       sqrt(-1)
       catch
       error("Encountered an error: [my debugging info]")
       end
ERROR: Encountered an error: [my debugging info]
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] top-level scope
   @ REPL[1]:4

caused by: DomainError with -1.0:
sqrt was called with a negative real argument but will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).
Stacktrace:
 [1] throw_complex_domainerror(f::Symbol, x::Float64)
   @ Base.Math ./math.jl:33
 [2] sqrt
   @ ./math.jl:686 [inlined]
 [3] sqrt(x::Int64)
   @ Base.Math ./math.jl:1578
 [4] top-level scope
   @ REPL[1]:2

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge May 1, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As to why I want to throw my own error, you're right, it's so that I can see the full URL easily.

If I just do the call to HTTP.head(), here's what I get:

julia> import HTTP

julia> HTTP.head("https://example.com/foo/bar/baz")
ERROR: HTTP.Exceptions.StatusError(404, "HEAD", "/foo/bar/baz", HTTP.Messages.Response:
"""
HTTP/1.1 404 Not Found
Date: Fri, 01 May 2026 02:29:11 GMT
Content-Type: text/html
Connection: keep-alive
Server: cloudflare
Age: 8057
cf-cache-status: HIT
Content-Encoding: gzip
CF-RAY: 9f4b5b6c49bbcc9b-BOS

""")
Stacktrace: [elided]

So the error only shows the path (/foo/bar/baz), and I have to go back and look in the code to remind myself what the host was, which is annoying.

etag = HTTP.header(response, "ETag", nothing)
etag === nothing && @warn "ETag not provided in response from $url"
last_modified = HTTP.header(response, "Last-Modified", nothing)
last_modified === nothing && @warn "Last-Modified not provided in response from $url"
return HeadInfo(; url=URI(url), etag, last_modified)
end

##### --------------------------------------------------------------------------------------

function main(out_path)
tags = get_tags()
tag_versions = filter(x -> x !== nothing, [vnum_maybe(basename(t["ref"])) for t in tags])
Expand Down Expand Up @@ -240,6 +266,8 @@ function main(out_path)

end

headinfo = HeadInfo(url)

# Build up metadata about this file
file_dict = Dict(
"triplet" => triplet(platform),
Expand All @@ -260,6 +288,13 @@ function main(out_path)
file_dict["asc"] = asc_signature
end

if !isnothing(headinfo.etag)
file_dict["etag"] = headinfo.etag
end
if !isnothing(headinfo.last_modified)
file_dict["last-modified"] = headinfo.last_modified
Copy link
Copy Markdown
Member

@ararslan ararslan Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ease of downstream comparison, should we parse this into a DateTime and write it out as ISO-8601?

If we want to handle all of the formats mentioned in RFC 9110, this should work:

let fmts = [dateformat"e, d u y H:M:S \G\M\T",  # IMF-fixdate (RFC 5322)
            dateformat"E, d-u-y H:M:S \G\M\T",  # RFC 850
            dateformat"e u d H:M:S y"]          # ANSI C asctime()
    global function parse_http_date(dt)
        dt = replace(dt, r"\s+" => " ")  # asctime left-pads days with space instead of 0
        for fmt in fmts
            x = tryparse(DateTime, dt, fmt)
            x !== nothing && return x
        end
        throw(ArgumentError("date is not in a recognized format: $dt"))
    end
end

But I think the RFC 5322 format is what we can expect to receive from any server that doesn't think it's currently 1994.

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge May 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I thought about parsing it, but I don't think we'll ever actually do date comparisons or arithmetic. I.e. my plan is to just see if the value of Last-Modified has changed, instead of comparing the timestamp to now().

end

# Right now, all we have are archives, but let's be forward-thinking
# and make this an array of dictionaries that is easy to extensibly match
push!(meta[version]["files"], file_dict)
Expand Down
2 changes: 2 additions & 0 deletions test/more_tests.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,10 @@ end
]
optional_keys = [
"asc",
"etag",
"git-tree-sha1",
"git-tree-sha256",
"last-modified",
]
allowed_keys = union(required_keys, optional_keys)
@test required_keys ⊆ collect(keys(filedict))
Expand Down
Loading