Description
ManifestFromRemote() in image/manifest.go resolves all layer blob URLs up front (lines 105–136) before any layer content is fetched. For each layer, it makes a Range: bytes=0-0 request, follows the redirect to the object storage presigned URL, and captures that URL in the manifest struct.
The actual layer downloading happens later when libindex.Index() is called with the pre-built manifest. Layers are fetched sequentially using the captured presigned URLs.
When an image contains large layers (e.g. multi-GB), downloading earlier layers can take long enough that the presigned URLs for later layers expire before they are ever requested. Object storage backends typically set X-Amz-Expires=600 (10 minutes) on these URLs.
Error
layers fetch failure error="fetcher: encountered errors: error realizing layer sha256:...: unexpected status code: 403 Forbidden
(body starts: \"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code>
<Message>Request has expired</Message><X-Amz-Expires>600</X-Amz-Expires>...\")"
Root Cause
image/manifest.go:ManifestFromRemote() captures presigned URLs eagerly for all layers before any downloading begins. The report command then passes this manifest to libindex.Index(), which fetches layers sequentially. By the time it reaches later layers, their presigned URLs have expired.
Suggested Fix
Defer URL resolution to fetch time — either by moving the redirect-following logic into a custom FetchArena/Realizer implementation, or by having the Realizer.Realize() method re-resolve any URL that is close to expiry before fetching.
Description
ManifestFromRemote()inimage/manifest.goresolves all layer blob URLs up front (lines 105–136) before any layer content is fetched. For each layer, it makes aRange: bytes=0-0request, follows the redirect to the object storage presigned URL, and captures that URL in the manifest struct.The actual layer downloading happens later when
libindex.Index()is called with the pre-built manifest. Layers are fetched sequentially using the captured presigned URLs.When an image contains large layers (e.g. multi-GB), downloading earlier layers can take long enough that the presigned URLs for later layers expire before they are ever requested. Object storage backends typically set
X-Amz-Expires=600(10 minutes) on these URLs.Error
Root Cause
image/manifest.go:ManifestFromRemote()captures presigned URLs eagerly for all layers before any downloading begins. Thereportcommand then passes this manifest tolibindex.Index(), which fetches layers sequentially. By the time it reaches later layers, their presigned URLs have expired.Suggested Fix
Defer URL resolution to fetch time — either by moving the redirect-following logic into a custom
FetchArena/Realizerimplementation, or by having theRealizer.Realize()method re-resolve any URL that is close to expiry before fetching.