rfc6249.xml

<?xml version="1.0" encoding="US-ASCII"?>

<?xml-stylesheet type='text/xsl' href='rfc2629.xslt' ?>
<?rfc toc="yes" ?>
<?rfc tocdepth="3" ?>
<?rfc tocindent="yes" ?>
<?rfc symrefs="yes" ?>
<?rfc sortrefs="yes"?>
<?rfc iprnotified="no" ?>
<?rfc compact="yes" ?>
<?rfc subcompact="no"?>
<?rfc rfcedstyle="yes"?>

<!DOCTYPE rfc
  PUBLIC "" "rfc2629.dtd">

<rfc category="std" number="6249" ipr="trust200902" submissionType="IETF" consensus="yes">

    <front>        
      <title abbrev="Metalink/HTTP: Mirrors and Hashes">Metalink/HTTP: Mirrors and Hashes</title>                
      <author initials="A." surname="Bryan" fullname="Anthony Bryan">
            <organization></organization>
            <address>
                  <postal>
                        <street></street>
                    <city>Pompano Beach</city>
                        <region>FL</region>
                        <country>USA</country>
                  </postal>
              <email>anthonybryan@gmail.com</email>        
              <uri>http://www.metalinker.org</uri>                
            </address>
      </author>

          <author initials="N." surname="McNab" fullname="Neil McNab">
        <organization></organization>
        <address>
          <email>neil@nabber.org</email>
          <uri>http://www.nabber.org</uri>
        </address>
      </author>

      <author initials="T." surname="Tsujikawa" fullname="Tatsuhiro Tsujikawa">
        <organization></organization>
        <address>
                  <postal>
                        <street></street>
                        <city></city>
                        <region>Shiga</region>
                        <country>Japan</country>
                  </postal>
          <email>tatsuhiro.t@gmail.com</email>
          <uri>http://aria2.sourceforge.net</uri>
        </address>
      </author>

      <author initials="P." surname="Poeml" fullname="Dr. med. Peter Poeml">
        <organization>MirrorBrain</organization>
        <address>
          <postal>
            <street>Venloer Str. 317</street>
            <city>Koeln</city>
            <code>50823</code>
            <country>DE</country>
          </postal>
          <phone>+49 221 6778 333 8</phone>
          <email>peter@poeml.de</email>
          <uri>http://mirrorbrain.org/~poeml/</uri>
        </address>
      </author>

      <author initials="H." surname="Nordstrom" fullname="Henrik Nordstrom">
        <organization></organization>
        <address>
          <email>henrik@henriknordstrom.net</email>
          <uri>http://www.henriknordstrom.net/</uri>
        </address>
      </author>
          <date month="May" year="2011"/>

<!-- [rfced] Please insert any keywords (beyond those that appear in the
title) for use on http://www.rfc-editor.org/rfcsearch.html. -->

      <keyword>file transfer</keyword>
      <keyword>download</keyword>
      <keyword>link</keyword>
      <keyword>signature</keyword>	  
      <keyword>mirrors</keyword>
      <keyword>data integrity</keyword>
      <keyword>hash</keyword>
      <keyword>http</keyword>
      <keyword>hypertext transfer protocol</keyword>
      <keyword>ftp</keyword>
      <keyword>file transfer protocol</keyword>
      <keyword>metadata</keyword>
      <keyword>torrent</keyword>

<!-- [rfced] The following terms may be used inconsistently throughout the document.  Please let us know if/how they may be made consistent.

range / Range
ranged / Ranged -->

      <abstract>
    <t>This document specifies Metalink/HTTP: Mirrors and Cryptographic Hashes in HTTP header fields, a different way to get information that is usually contained in the Metalink XML-based download description format. 
    Metalink/HTTP describes multiple download locations (mirrors), Peer-to-Peer, cryptographic hashes, digital signatures, and other information using existing standards for HTTP header fields. Metalink clients can use this information to make file transfers more robust and reliable.
        Normative requirements for Metalink/HTTP clients and servers are described here.</t>
      </abstract>
  </front>    

  <middle>
    <section title="Introduction">
      
      <t>Metalink/HTTP is an alternative and complementary representation of Metalink information, which is usually presented as an XML-based document format <xref target="RFC5854"/>. 
      Metalink/HTTP attempts to provide as much functionality as the Metalink/XML format by using existing standards, such as Web Linking <xref target="RFC5988"/>, 
      Instance Digests in HTTP <xref target="RFC3230"/>, and Entity Tags (also known as ETags) <xref target="RFC2616"/>. Metalink/HTTP is used to list
      information about a file to be downloaded. This can include lists of multiple URIs (mirrors), Peer-to-Peer information, cryptographic hashes, and digital signatures.</t>
      
      <t>Identical copies of a file are frequently accessible in multiple locations on the Internet over a variety of protocols (such as FTP, HTTP, and Peer-to-Peer).
      In some cases, users are shown a list of these multiple download locations (mirrors) and must manually select a single one on the basis of geographical location, priority, or bandwidth.
      This distributes the load across multiple servers, and should also increase throughput and resilience. At times, however, individual servers can be slow, outdated, or unreachable, but this cannot be determined until the download has been initiated. Users will rarely have sufficient information to choose the most appropriate server and will often choose the first in a list, which might not be optimal for their needs, and will lead to a particular server getting a disproportionate share of load.
      The use of suboptimal mirrors can lead to the user canceling and restarting the download to try to manually find a better source. During downloads, errors in transmission can corrupt the file.
      There are no easy ways to repair these files. For large downloads, this can be extremely troublesome.
      Any of the number of problems that can occur during a download lead to frustration on the part of users.</t>

      <t>Some popular sites automate the process of selecting mirrors using DNS load balancing, both to approximately balance load between servers, and to direct clients to nearby servers with the hope that this improves throughput.  Indeed, DNS load balancing can balance long-term server load fairly effectively, but it is less effective at delivering the best throughput to users when the bottleneck is not the server but the network.</t>

      <t>This document describes a mechanism by which the benefit of mirrors can be automatically and more effectively realized. All the information about a download, including mirrors, cryptographic hashes, digital signatures, and more can be transferred in coordinated HTTP header fields, hereafter referred to as a "Metalink".
      This Metalink transfers the knowledge of the download server (and mirror
database) to the client. Clients can fall back to other mirrors if the current one has an issue. With this knowledge, 
      the client is enabled to work its way to a successful download even under adverse circumstances. 
      All this can be done without complicated user interaction, and the download can be much more reliable and efficient.
      In contrast, a traditional HTTP redirect to a mirror conveys only minimal
information -- one link to one server -- and there is no provision in the HTTP protocol to handle failures.
      Furthermore, in order to provide better load distribution across servers and potentially faster downloads to users, Metalink/HTTP facilitates multi-source downloads, where portions of a file are downloaded from multiple mirrors (and, optionally, Peer-to-Peer) simultaneously.</t>

          <t>Upon connection to a Metalink/HTTP server, a client will receive information about other sources of the same resource and a cryptographic hash of the whole resource. The client will then be able to request chunks of the file from the various sources, scheduling appropriately in order to maximize the download rate.</t>
   
      <section title="Example Metalink Server Response"><t><figure> 
          <preamble>This example shows a brief Metalink server response with ETag, mirrors, Peer-to-Peer information, Metalink/XML, OpenPGP signature, and a cryptographic hash of the whole file:</preamble>
          
          <artwork type="example">
Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: &lt;http://www2.example.com/example.ext&gt;; rel=duplicate
Link: &lt;ftp://ftp.example.com/example.ext&gt;; rel=duplicate
Link: &lt;http://example.com/example.ext.torrent&gt;; rel=describedby;
type="application/x-bittorrent"
Link: &lt;http://example.com/example.ext.meta4&gt;; rel=describedby;
type="application/metalink4+xml"
Link: &lt;http://example.com/example.ext.asc&gt;; rel=describedby;
type="application/pgp-signature"
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==
</artwork></figure></t>
      </section> 
      <section title="Notational Conventions"> 

        <t>This specification describes conformance of Metalink/HTTP.</t>
     

        <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
        "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
        document are to be interpreted as described in BCP 14, <xref target="RFC2119"/>, as scoped to those conformance targets.</t>

      </section>
          <section title="Terminology">
                    <t>The following terms, as used in this document, are defined here:
            <list style="symbols">

                  <t>Metalink server: HTTP server that provides a Metalink in HTTP response header fields.</t>
              <t>Metalink : A collection of HTTP response header fields from a
Metalink server, which is the reply to a GET or HEAD request from a client and which includes Link header fields listing mirrors and Instance Digests listing a cryptographic hash.</t>
                  <t>Link header field: HTTP response header field, defined in <xref target="RFC5988"/>, that can list mirrors and, potentially, other download methods for obtaining a file, along with digital signatures.</t>
                  <t>Instance Digest: HTTP response header field, defined in <xref target="RFC3230"/>, that contains the cryptographic hash of a file, which is used by the Metalink client to verify the integrity of the file once the download has completed.</t>
                  <t>Entity Tag or ETag: HTTP response header field, defined in
<xref target="RFC2616"/>, that, if synchronized between the Metalink server and mirror servers, allows Metalink clients to provide advanced features.</t>
                  <t>Mirror server: Typically, FTP or HTTP servers that
"mirror" the Metalink server, i.e., provide identical copies of (at least some) files that are also on the mirrored server.</t>
                  <t>Metalink clients: Applications that process Metalinks and
use them to provide an improved download experience. They support HTTP and could also support other download protocols like FTP or various Peer-to-Peer methods.</t>
                  <t>Metalink/XML: An XML file that can contain similar information to an HTTP response header Metalink, such as mirrors and cryptographic hashes.</t>
            </list></t>          
          </section>
    </section>

    <section title="Requirements" anchor="req">
      <t>In this context, "Metalink" refers to Metalink/HTTP, which consists of mirrors and cryptographic hashes in HTTP header fields as described in this document. "Metalink/XML" refers to the XML format described in <xref target="RFC5854"/>.</t>
          
      <t>Metalink resources include Link header fields <xref target="RFC5988"/> to present a list of mirrors in the response to a client request for the resource. Metalink servers MUST include the cryptographic hash of a resource via Instance Digests in HTTP <xref target="RFC3230"/>.
          Algorithms used in the Instance Digest field are registered in the
IANA registry named "Hypertext Transfer Protocol (HTTP) Digest Algorithm
Values" at
&lt;http://www.iana.org/&gt;. This document restricts the use of these
algorithms.

<!-- [rfced] Sections 2 and 8:  Please note that the IANA URLs have
been shortened per Section 5.1 of RFC 5226.

Original (Section 2):
 Algorithms used in the
 Instance Digest field are registered in the IANA registry named
 "Hypertext Transfer Protocol (HTTP) Digest Algorithm Values" at
 <http://www.iana.org/assignments/http-dig-alg/http-dig-alg.xhtml>.

Changed to:
 Algorithms used in the
 Instance Digest field are registered in the IANA registry named
 "Hypertext Transfer Protocol (HTTP) Digest Algorithm Values" at
 <http://www.iana.org/>. -->

          SHA-256 and SHA-512 were added to the registry by <xref
target="RFC5843"/>.  Metalinks contain whole file hashes as described in <xref
target="wholehashes"/>, and MUST include SHA-256, as specified in <xref
target="FIPS-180-3"/>.  It MAY also include other hashes.</t>

          
          <t>Metalink servers are HTTP servers with one or more Metalink
 resources. Metalink servers MUST support the Link header fields for listing mirrors and MUST support Instance Digests in HTTP <xref target="RFC3230"/>. Metalink servers MUST return the same Link header fields and Instance Digests on HEAD requests. 
      Metalink servers and their associated preferred
mirror servers MUST all share the same ETag policy.
Metalink servers and their associated normal
mirror servers SHOULD all share the same ETag policy.
(See <xref target="coordinatedpolicies"/> for the definition of "preferred" and "normal" mirror servers.) It is up to the administrator of the Metalink server to communicate the details of the shared ETag policy to the administrators of the mirror servers so that the mirror servers can be configured with the same ETag policy. To have the same ETag policy means that ETags are synchronized across servers for resources that are mirrored; i.e., byte-for-byte identical files will have the same ETag on mirrors that they have on the Metalink server.   For example, it would be better to derive an ETag from a
  cryptographic hash of the file contents than on server-unique filesystem metadata. Metalink servers SHOULD offer Metalink/XML documents that contain cryptographic hashes of parts of the file (and other information) if error recovery is desirable.</t>

      <t>Mirror servers are typically FTP or HTTP servers that "mirror" another server. That is, they provide identical copies of (at least some) files that are also on the mirrored server.
      Mirror servers SHOULD support serving partial content. HTTP mirror servers SHOULD share the same ETag policy as the originating Metalink server. HTTP mirror servers SHOULD support Instance Digests in HTTP <xref target="RFC3230"/> using the same algorithm as the Metalink server.  Optimally, HTTP mirror servers will share the same ETag policy and support Instance Digests in HTTP.
          Mirror servers that share the same ETag policy and/or support Instance Digests in HTTP using the same algorithm as a Metalink server are known as preferred mirror servers.</t>

      <t>Metalink clients use the mirrors provided by a Metalink server in Link
header fields <xref target="RFC5988"/> but these clients are restricted to
using the mirrors provided by the initial Metalink server they contacted.
 If Metalink clients find Link header fields
<xref target="RFC5988"/> from mirrors that in turn list mirrors, or from a Metalink server
 listing itself as a mirror, they MUST discard such Link header
 fields <xref target="RFC5988"/> to prevent a possible infinite loop.

<!-- [rfced] May we clarify the following sentence?  It is unclear to what "it" refers in the first sentence.  In the second sentence, perhaps "for listing mirrors from mirrors" will be confusing.

Original:
 Metalink clients use the mirrors provided by a Metalink server in
 Link header fields [RFC5988] but it is restricted to the initial
 Metalink server they contacted.  If Metalink clients find Link header
 fields [RFC5988] for listing mirrors from mirrors or a Metalink
 server listing itself as a mirror, they MUST discard such Link header
 fields [RFC5988] to prevent a possible infinite loop.

Suggested:
 Metalink clients use the mirrors provided by a Metalink server in
 Link header fields [RFC5988], but these clients are restricted to
 using the mirrors provided by the initial Metalink server they
 contacted.  If Metalink clients find Link header fields [RFC5988]
 from mirrors that in turn list mirrors, or from a Metalink server
 listing itself as a mirror, they MUST discard such Link header
 fields [RFC5988] to prevent a possible infinite loop. -->

Metalink clients MUST
support HTTP and SHOULD support FTP <xref target="RFC0959"/>. Metalink clients
MAY support BitTorrent <xref target="BITTORRENT"/> 
      or other download methods. Metalink clients SHOULD switch downloads from
one mirror to another if a mirror becomes unreachable. Metalink clients MAY
support multi-source, or parallel, 
      downloads, where portions of a file can be downloaded from multiple
mirrors simultaneously (and, optionally, from Peer-to-Peer sources). Metalink
clients MUST support Instance Digests in HTTP <xref target="RFC3230"/> by 
      requesting and verifying cryptographic hashes. Metalink clients SHOULD
support error recovery by using the cryptographic hashes of parts of the file
listed in Metalink/XML files. Metalink clients SHOULD support checking digital
signatures.</t>
          
          </section>
        
    <section title="Mirrors / Multiple Download Locations" anchor="mirrors">
      <t>Mirrors are specified with the Link header fields <xref target="RFC5988"/> and a relation type of "duplicate" as defined in <xref target="IANA"/>.</t>
          <t>The following list contains OPTIONAL attributes, which are defined elsewhere in this document:
            <list style="symbols">

              <t>"depth" : mirror depth (see <xref target="depth"/>).</t>
              <t>"geo"   : mirror geographical location (see <xref target="geo"/>).</t>
              <t>"pref"  : a preferred mirror server (see <xref target="coordinatedpolicies"/>).</t>
              <t>"pri"   : mirror priority (see <xref target="priority"/>).</t>
            </list></t>          

<t><figure> 
          <preamble>This example shows a brief Metalink server response with two mirrors only:</preamble>
          <artwork type="example">
Link: &lt;http://www2.example.com/example.ext&gt;; rel=duplicate; 
pri=1; pref
Link: &lt;ftp://ftp.example.com/example.ext&gt;; rel=duplicate; 
pri=2; geo=gb; depth=1
</artwork></figure></t>

      <t>As some organizations can have many mirrors, it is up to the organization to configure the amount of Link header fields the Metalink server will provide. Such a decision could be a random selection or a hard-coded limit based on network proximity, file size, server load, or other factors.</t>

<section title="Mirror Priority" anchor="priority">
      <t>Entries for mirror servers MAY have a "pri" value to designate the priority of a mirror. Valid ranges for the "pri" attribute are from 1 to 999999. Mirror servers with a lower value of the "pri" attribute have a higher priority, while mirrors with an undefined "pri" attribute are considered to have a value of 999999, which is the lowest priority. For example, a mirror with "pri=10"
      has higher priority than a mirror with "pri=20".
          Metalink clients SHOULD use mirrors with lower "pri" values first, but depending on other conditions, they MAY decide to use other mirrors.</t>
          
      <t>This is purely an expression of the server's preferences; it is up to the client what it does with this information, particularly with reference to how many servers to use at any one time.</t>

</section>

<section title="Mirror Geographical Location" anchor="geo">
        <t>Entries for a mirror server MAY have a "geo" value, which
        is an <xref target="ISO3166-1"/> alpha-2 two-letter country code for the geographical location of the physical server the URI is used
        to access. A client MAY use this information to select a mirror, or set
of mirrors, that is geographically near (if the client has access to such
information), with the aim of reducing network load at inter-country
bottlenecks.
        </t>
</section>

<section title="Coordinated Mirror Policies" anchor="coordinatedpolicies">

      <t>There are two types of mirror servers: preferred and normal.
Entries for preferred HTTP mirror servers have a "pref" value
and entries for normal mirrors don't.
Preferred mirror servers are HTTP mirror servers that MUST share the
same ETag policy as the originating Metalink server and/or MUST
provide Instance Digests using the same algorithm as the Metalink
server. Preferred mirrors make it possible for Metalink clients to
detect early on, before data is transferred, if the file requested
matches the desired file. This early file mismatch detection is
described in <xref target="earlyfilemismatchdetection"/>.
Normal mirrors do not necessarily share the same ETag
policy or support Instance Digests using the same algorithm as the
Metalink server. FTP mirrors are considered "normal", as they do not
emit ETags or support Instance Digests.</t>
          
</section>

<section title="Mirror Depth" anchor="depth">
      <t>Some mirrors can mirror single files, whole directories, or multiple directories.</t>
      <t>Entries for mirror servers can have a "depth" value, where "depth=0" is the default. A value of 0 means that only that file is mirrored and that other URI path segments are not. A value of 1 means that the file and all other files and URI path segments contained in the
rightmost URI path segment are mirrored. For values of N, N-1 URI path segments closer to the Host are mirrored. A value of 2 means one URI path segment closer to the Host is mirrored, and all files and URI path segments contained are mirrored. 
For each higher value, another URI path segment closer to the Host is mirrored.</t>

<t><figure> 
          <preamble>This example shows a mirror with a depth value of 4:</preamble>          
          <artwork type="example">
Link: &lt;http://www2.example.com/dir1/dir2/dir3/dir4/dir5/example.ext&gt;; 
rel=duplicate; pri=1; pref; depth=4
</artwork></figure></t>

      <t>In the above example, four URI path segments closer to the Host are mirrored, from /dir2/ and all files and directories included.</t>

</section>

          </section>

    <section title="Peer-to-Peer / Metainfo" anchor="p2p">
      <t>Entries for metainfo files, which describe ways to download a file over Peer-to-Peer networks or otherwise, are specified with the Link header fields <xref target="RFC5988"/> and a relation type of "describedby" and a type parameter that indicates
   the MIME type of the metadata available at the URI. Since metainfo files can sometimes describe multiple files, or the filename MAY not be the same on the Metalink server and in the metainfo file but MAY still have the same content, an OPTIONAL "name" attribute can be used.</t>

          <t>The following list contains an OPTIONAL attribute, which is defined in this document:
            <list style="symbols">

                  <t>"name" : a file described within the metainfo file.</t>
            </list></t>          
   
<t><figure> 
          <preamble>This example shows a brief Metalink server response with .torrent and .meta4:</preamble>        
          <artwork type="example">
Link: &lt;http://example.com/example.ext.torrent&gt;; rel=describedby;
type="application/x-bittorrent"; name="differentname.ext"
Link: &lt;http://example.com/example.ext.meta4&gt;; rel=describedby;
type="application/metalink4+xml"
</artwork></figure></t>

<t>Metalink clients MAY support the use of metainfo files for downloading files.</t>

    <section title="Metalink/XML Files" anchor="metalinkxml">
      <t>Metalink/XML files for a given resource MAY be provided in a Link header field as shown in the example in <xref target="p2p"/>.
          Metalink/XML files are specified in <xref target="RFC5854"/>, and they are particularly useful for providing metadata such as cryptographic hashes of parts of a file, allowing a
      client to recover from errors (see <xref target="errorcorrection"/>). 
Metalink servers SHOULD provide Metalink/XML files with partial file hashes in
Link header fields, and Metalink clients SHOULD use them for error recovery.</t>
    </section>
        
    </section>

<section title="Signatures" anchor="signatures">
        
    <section title="OpenPGP Signatures" anchor="openpgp">
      <t>OpenPGP signatures <xref target="RFC3156"/> of requested files are specified with the Link header fields <xref target="RFC5988"/> and a relation type of "describedby" and a type parameter of "application/pgp-signature".</t>
<t><figure> 
          <preamble>This example shows a brief Metalink server response with OpenPGP signature only:</preamble>
          <artwork type="example">
Link: &lt;http://example.com/example.ext.asc&gt;; rel=describedby;
type="application/pgp-signature"
</artwork></figure></t>
    
<t>Metalink clients SHOULD support the use of OpenPGP signatures.</t>
    </section>
        
    <section title="S/MIME Signatures">
      <t>Secure/Multipurpose Internet Mail Extensions (S/MIME)
signatures <xref target="RFC5751"/> of requested files are specified with the Link header fields <xref target="RFC5988"/> and a relation type of "describedby" and a type parameter of "application/pkcs7-mime".</t>
<t><figure> 
          <preamble>This example shows a brief Metalink server response with S/MIME signature only:</preamble>
          <artwork type="example">
Link: &lt;http://example.com/example.ext.p7m&gt;; rel=describedby;
type="application/pkcs7-mime"
</artwork></figure></t>
    
<t>Metalink clients SHOULD support the use of S/MIME signatures.</t>
</section>
</section>
        
    <section title="Cryptographic Hashes of Whole Documents" anchor="wholehashes">
        
        <t>If Instance Digests are not provided by the Metalink servers, the Link header fields pertaining to this specification MUST be ignored.</t>
        
<t><figure> 
          <preamble>This example shows a brief Metalink server response with ETag, mirror, and cryptographic hash:</preamble>        
          <artwork type="example">
Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: &lt;http://www2.example.com/example.ext&gt;; rel=duplicate
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==
</artwork></figure></t>


    </section>

    <section title="Client / Server Multi-Source Download Interaction" anchor="clientserver">
      <t>Metalink clients begin a download with a standard HTTP <xref target="RFC2616"/> GET request to the Metalink server. Metalink clients MAY use a range limit if desired.</t>
<t><figure> 
          <artwork type="example">GET /distribution/example.ext HTTP/1.1
Host: www.example.com
</artwork></figure></t>

<t>The Metalink server responds with the data and these header fields:

<figure> 
          <artwork type="example">
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 14867603
Content-Type: application/x-cd-image
Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Link: &lt;http://www2.example.com/example.ext&gt;; rel=duplicate; pref
Link: &lt;ftp://ftp.example.com/example.ext&gt;; rel=duplicate
Link: &lt;http://example.com/example.ext.torrent&gt;; rel=describedby;
type="application/x-bittorrent"
Link: &lt;http://example.com/example.ext.meta4&gt;; rel=describedby;
type="application/metalink4+xml"
Link: &lt;http://example.com/example.ext.asc&gt;; rel=describedby;
type="application/pgp-signature"
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==
</artwork></figure></t>

<t>Alternatively, Metalink clients can begin with a HEAD request to the Metalink server to discover mirrors via Link header fields
 and then skip to making the following decisions on every available mirror server found via the Link header fields.</t>

<t>After that, the client follows with a GET request to the desired mirrors.</t>

<t>From the Metalink server response, the client learns some or all of the following metadata
about the requested object, in addition to starting to receive the
object:</t>
<t>
<list style="symbols">
<t>Mirror profile link, which can describe the mirror's priority, whether it shares the ETag policy of the originating Metalink server, geographical location, and mirror depth.</t>
<t>Instance Digest, which is the whole file cryptographic hash.</t>
<t>ETag.</t>
<t>Object size from the Content-Length header field.</t>
<t>Metalink/XML, which can include partial file cryptographic hashes to repair a file.</t>
<t>Peer-to-Peer information.</t>
<t>Digital signature.</t>
</list></t>

<t>Next, the Metalink client requests a range of the object from a preferred mirror server, so it can use If-Match conditions:

<figure> 
          <artwork type="example">
GET /example.ext HTTP/1.1
Host: www2.example.com
Range: bytes=7433802-
If-Match: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Referer: http://www.example.com/distribution/example.ext
</artwork></figure></t>

<t>Metalink clients SHOULD use preferred mirrors, if possible, as they allow early file mismatch detection as described in <xref target="earlyfilemismatchdetection"/>.
Preferred mirrors have coordinated ETags, as described in <xref target="coordinatedpolicies"/>, and Metalink clients SHOULD use If-Match conditions based on the ETag to quickly 
detect out-of-date mirrors by using the ETag from the Metalink server response. Metalink clients SHOULD use partial file cryptographic hashes as described in <xref target="errorcorrection"/>, if available, to detect if the mirror server returned the correct data.</t>

<t>Optimally, the mirror server also will include an Instance Digest in the mirror response to the client GET request, which the client can also use to detect a mismatch early. 
Metalink clients MUST reject individual downloads from mirrors that support Instance Digests if the Instance Digest from the mirror does not match the Instance Digest as reported by the Metalink server and the same algorithm is used.
If normal mirrors are used, then a mismatch cannot be 
detected until the completed object is verified. Errors in transmission and substitutions of incorrect data on mirrors, whether deliberate or accidental, can be detected with error correction as described in <xref target="errorcorrection"/>.</t>

<t>Here, the preferred mirror server has the correct file (the If-Match conditions match) and responds with a 206 Partial Content HTTP status code and appropriate "Content-Length", "Content-Range", ETag, and Instance Digest header fields. In this example, the mirror server responds, with data, to the above request:

<figure> 
          <artwork type="example">
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Content-Length: 7433801
Content-Range: bytes 7433802-14867602/14867603
Etag: "thvDyvhfIqlvFe+A9MYgxAfm1q5="
Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==
</artwork></figure></t>

<t>
  Metalink clients MAY start a number of parallel range requests (one
  per selected mirror server other than the first) using mirrors
  provided by the Link header fields with "duplicate" relation type.
  Metalink clients MUST limit the number of parallel connections to
  mirror servers, ideally based on observing how the aggregate
  throughput changes as connections are opened.  It would be pointless
  to blindly open connections once the path bottleneck is filled.
  After establishing a new connection, a Metalink client SHOULD monitor
  whether the aggregate throughput increases over all connections that
  are part of the download. The client SHOULD NOT open additional
  connections during this period. If the aggregate throughput has
  increased, the client MAY open an additional connection and repeat
  these steps. Otherwise, the client SHOULD NOT open a new connection
  until an established one closes.
  Metalink clients SHOULD use the location of the original GET request
  in the "Referer" header field for these range requests.
</t>

<t>The Metalink client can determine the size and number of ranges requested from each server, based upon the type and number of mirrors and performance observed from each mirror.
Note that range requests impose an overhead on servers, and clients need to be aware of that and not abuse them. When downloading a particular file, Metalink clients MUST NOT make more than one
concurrent request to each mirror server from which it downloads.</t>

<t>Metalink clients SHOULD close all but the fastest connection if any range requests generated after the first request end up with a complete response, instead of a partial response
(as some mirrors might not support HTTP ranges), if the goal is the fastest transfer. Metalink clients MAY monitor mirror conditions and dynamically switch between mirrors to achieve the fastest download possible. Similarly, Metalink clients SHOULD abort extremely slow or stalled range requests and finish the request on
other mirrors. If all ranges have finished except for the final one, the Metalink client can split the final range into multiple range requests to other mirrors so the transfer finishes faster.</t>

<t>If the first request was a GET, no Range header field was sent, and
 the client determines later that it will issue a range request, then the client
 SHOULD close the first connection when it catches up with the other parallel range requests of the same object. This means the first connection was
 sacrificed. Metalink clients can use a HEAD request first, if
 possible, so that the client can find out if there are any Link
 header fields, and then range-based requests are undertaken to
 the mirror servers without sacrificing a first connection.</t>

<t>Metalink clients MUST reject individual downloads from mirrors where the file size does not match the file size as reported by the Metalink server.</t>

<t>If a Metalink client does not
support certain download methods (such as FTP or BitTorrent) that a
file is available from, and there are no available download methods
that the client supports, then the download will have no way to
complete.</t>

<t>Metalink clients MUST verify the cryptographic hash of the file once the download has completed. If the cryptographic hash offered by the Metalink server with Instance Digests
does not match the cryptographic hash
of the downloaded file, see <xref target="errorcorrection"/> for a possible way to repair errors.</t>

<t>If the download cannot be repaired, it is considered corrupt. The client can attempt to re-download the file.</t>

<t>Metalink clients that support verifying digital signatures MUST verify digital signatures of requested files if they are included.
Digital signatures MUST validate back to a trust anchor as described in the validation rules in <xref target="RFC3156"/> and <xref target="RFC5280"/>.</t>

<section title="Error Prevention, Detection, and Correction">

<t>Error prevention, or early file mismatch detection, is possible before file transfers with the use of file sizes, ETags, and Instance Digests provided by Metalink servers. Error detection requires Instance Digests
to detect errors in transfer after the transfers have completed. Error correction, or download repair, is possible with partial file cryptographic hashes.</t>

<t>Note that cryptographic hashes obtained from Instance Digests are in base64 encoding, while those from Metalink/XML are in hexadecimal.</t>

<section title="Error Prevention (Early File Mismatch Detection)" anchor="earlyfilemismatchdetection">

<t> In HTTP terms, the merging of ranges from multiple responses SHOULD
 be verified with a strong validator, which in this context is either
 an Instance Digest or a shared ETag from that Metalink server that
 matches with the Instance Digest or ETag provided by a preferred
 mirror server.

<!-- [rfced] Section 7.1.1:  Does "same" refer to "Instance Digest,"
"ETag," or both?

Original:
 In HTTP terms, the merging of ranges from multiple responses SHOULD
 be verified with a strong validator, which in this context is either
 an Instance Digest or a shared ETag from that Metalink server that
 matches with the same provided by a preferred mirror server.

Perhaps (if both):
 In HTTP terms, the merging of ranges from multiple responses SHOULD
 be verified with a strong validator, which in this context is either
 an Instance Digest or a shared ETag from that Metalink server that
 matches with the Instance Digest or ETag provided by a preferred
 mirror server. -->

In most cases, it is sufficient that the Metalink server provides
mirrors and Instance Digest information, but operation will be
more robust and efficient if the mirror servers do implement a shared
ETag policy or Instance Digests as well. There is no need to specify how the ETag is generated, just that it needs to be shared between the Metalink server and the mirror servers.
The benefit of having mirror servers return an Instance Digest is that the client then can detect
mismatches early even if ETags are not used. Mirrors that support both a shared ETag and Instance Digests do provide value, but just one is sufficient for early
detection of mismatches. If the mirror server provides neither shared ETag nor Instance Digest, then early detection of
mismatches is not possible unless file length also differs. Finally, errors are still detectable after the download has completed, when the cryptographic hash of the merged response is verified.</t>

<t>ETags cannot be used for verifying the integrity of the received
content. If the ETag given by the mirror server matches
the ETag given by the Metalink server, then the Metalink client assumes the responses are valid for that object.</t>

<t>This guarantees that a mismatch will be detected by using only the shared
ETag from a Metalink server and mirror server.
Metalink clients will detect an error if ETags do not match, which will
prevent accidental merges of ranges from different versions of files
with the same name.</t>

<t>A shared ETag or Instance Digest cannot strictly protect against malicious attacks or server or
network errors replacing content.
An attacker can make a mirror server seemingly respond with the expected Instance Digest or ETags even if the file
contents have been modified. The same goes
for various system failures, which would also cause bad data (i.e., corrupted files) to be returned. The Metalink client
has to rely on the Instance Digest returned by the Metalink server in the
first response for the verification of the downloaded object as a whole. To verify the individual ranges, which might have been requested from different sources, see <xref target="errorcorrection"/>.</t>

</section>

<section title="Error Correction" anchor="errorcorrection">

<t>Partial file cryptographic hashes can be used to detect errors during the download. Metalink servers SHOULD provide Metalink/XML files with partial file hashes 
in Link header fields as specified in <xref target="metalinkxml"/>, and Metalink clients SHOULD use them for error correction.</t>

<t>An error in transfer or a substitution attack will be detected by a cryptographic hash of the object not matching the Instance Digest from the Metalink server.
If the cryptographic hash of the object does not match the Instance Digest from the Metalink server, then the client SHOULD fetch the Metalink/XML (if available).
 This may contain partial file cryptographic hashes, which will allow detection of which mirror server returned incorrect data.  
 Metalink clients SHOULD use the Metalink/XML data to figure out what ranges of the downloaded data can be recovered and what needs to be fetched again. 
</t>

<t>Other methods can be used for error correction. For example, some other metainfo files also include partial file hashes that can be used to check for errors.</t>

</section>

</section>

    </section>

        <section title="IANA Considerations" anchor="IANA">

        <t>Accordingly, IANA has made the following registration to the "Link Relation Types" registry at &lt;http://www.iana.org/&gt;.</t>

<list style="symbols">
<t>Relation Name: duplicate</t>
<t>Description: Refers to a resource whose available representations are
byte-for-byte identical with the corresponding representations of the context
IRI.</t>
<t>Reference: This specification.</t>
<t>Notes: This relation is for static resources. That is, an HTTP GET request on any duplicate will return the same representation. It does not make sense for dynamic or POSTable resources and should not be used for them.</t></list>

        </section>

          <section title="Security Considerations">

            <section title="URIs and IRIs">

              <t>Metalink clients handle URIs and Internationalized Resource Identifiers (IRIs). See Section 7 of <xref target="RFC3986"/> and Section 8 of <xref target="RFC3987"/> for security
          considerations related to their handling and use.</t>

            </section>

            <section title="Spoofing">

              <t>There is potential for spoofing attacks where the attacker publishes Metalinks with false information.
          In that case, this could deceive unaware downloaders into downloading a malicious or worthless file. 
                  Metalink clients are advised to prevent loops, possibly from a mirror server to a Metalink server and back again, in <xref target="req"/>.
                  As with all downloads, users should only download from trusted sources. 
                  Also, malicious publishers could attempt a distributed
denial-of-service attack by inserting unrelated URIs into Metalinks. 
                  <xref target="RFC4732"/> contains information on amplification attacks and denial-of-service attacks.</t>

            </section>        
                
            <section title="Cryptographic Hashes">

              <t>Currently, some of the digest values defined in Instance Digests in HTTP <xref target="RFC3230"/> are considered insecure. 
          These include the whole Message Digest family of algorithms, which are not suitable for cryptographically strong verification. Malicious people could provide files that appear to be 
          identical to another file because of a collision; i.e., the weak cryptographic hashes of the intended file and a substituted malicious file could match.</t>

            </section>        
                
                
            <section title="Signing">

              <t>Metalinks SHOULD include digital signatures, as described in <xref target="signatures"/>.</t>

              <t>Digital signatures provide authentication and message integrity, and enable non-repudiation with proof of origin.</t>

            </section>

          </section>
    
        </middle> 

        <back> 
          
          <references title="Normative References">

<reference anchor="BITTORRENT" target="http://www.bittorrent.org/beps/bep_0003.html">
<front>
<title>The BitTorrent Protocol Specification</title>

<author initials="B" surname="Cohen" fullname="Bram Cohen">
    <organization/>
</author>


<date month="February" day="28" year="2008"/>
</front>

<seriesInfo name="BITTORRENT" value="11031"/>
</reference>        

<reference anchor="RFC0959">

<front>
<title>File Transfer Protocol</title>
<author initials="J." surname="Postel" fullname="J. Postel">
<organization/></author>
<author initials="J." surname="Reynolds" fullname="J. Reynolds">
<organization/></author>
<date year="1985" month="October"/></front>

<seriesInfo name="STD" value="9"/>
<seriesInfo name="RFC" value="0959"/>
</reference>

<reference anchor="RFC2119">
<front>
<title abbrev="RFC Key Words">Key words for use in RFCs to Indicate Requirement Levels</title>
<author initials="S." surname="Bradner" fullname="Scott Bradner"></author>
<date year="1997" month="March"/>
</front>
<seriesInfo name="BCP" value="14"/>
<seriesInfo name="RFC" value="2119"/>
</reference>

<reference anchor="RFC2616">
<front>
<title>Hypertext Transfer Protocol -- HTTP/1.1</title>
<author initials="R." surname="Fielding" fullname="R. Fielding">
<organization/></author>
<author initials="J." surname="Gettys" fullname="J. Gettys">
<organization/></author>
<author initials="J." surname="Mogul" fullname="J. Mogul">
<organization/></author>
<author initials="H." surname="Frystyk" fullname="H. Frystyk">
<organization/></author>
<author initials="L." surname="Masinter" fullname="L. Masinter">
<organization/></author>
<author initials="P." surname="Leach" fullname="P. Leach">
<organization/></author>
<author initials="T." surname="Berners-Lee" fullname="T. Berners-Lee">
<organization/></author>
<date year="1999" month="June"/></front>
<seriesInfo name="RFC" value="2616"/>
</reference>

<reference anchor='RFC3156'>
<front>
<title>MIME Security with OpenPGP</title>
<author initials='M.' surname='Elkins' fullname='M. Elkins'>
<organization /></author>
<author initials='D.' surname='Del Torto' fullname='D. Del Torto'>
<organization /></author>
<author initials='R.' surname='Levien' fullname='R. Levien'>
<organization /></author>
<author initials='T.' surname='Roessler' fullname='T. Roessler'>
<organization /></author>
<date year='2001' month='August' />
</front>
<seriesInfo name='RFC' value='3156' />
</reference>

<reference anchor="RFC3230">
<front>
<title>Instance Digests in HTTP</title>
<author initials="J." surname="Mogul" fullname="J. Mogul">
<organization/></author>
<author initials="A." surname="Van Hoff" fullname="A. Van Hoff">
<organization/></author>
<date year="2002" month="January"/></front>
<seriesInfo name="RFC" value="3230"/>
</reference> 

<reference anchor="RFC3986">
<front>
<title>Uniform Resource Identifier (URI): Generic Syntax</title>
<author initials="T." surname="Berners-Lee" fullname="T. Berners-Lee">
<organization/></author>
<author initials="R." surname="Fielding" fullname="R. Fielding">
<organization/></author>
<author initials="L." surname="Masinter" fullname="L. Masinter">
<organization/></author>
<date year="2005" month="January"/></front>
<seriesInfo name="STD" value="66"/>
<seriesInfo name="RFC" value="3986"/>
</reference>
 
<reference anchor="RFC3987">
<front>
<title>Internationalized Resource Identifiers (IRIs)</title>
<author initials="M." surname="Duerst" fullname="M. Duerst">
<organization/></author>
<author initials="M." surname="Suignard" fullname="M. Suignard">
<organization/></author>
<date year="2005" month="January"/></front>
<seriesInfo name="RFC" value="3987"/>
</reference>

<reference anchor='RFC5280'>
<front>
<title>Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile</title>
<author initials='D.' surname='Cooper' fullname='D. Cooper'>
<organization /></author>
<author initials='S.' surname='Santesson' fullname='S. Santesson'>
<organization /></author>
<author initials='S.' surname='Farrell' fullname='S. Farrell'>
<organization /></author>
<author initials='S.' surname='Boeyen' fullname='S. Boeyen'>
<organization /></author>
<author initials='R.' surname='Housley' fullname='R. Housley'>
<organization /></author>
<author initials='W.' surname='Polk' fullname='W. Polk'>
<organization /></author>
<date year='2008' month='May' />
</front>
<seriesInfo name='RFC' value='5280' />
</reference>

<reference anchor="RFC5988">
  <front>
    <title abbrev="Web Linking">Web Linking</title>
    <author initials="M." surname="Nottingham" fullname="Mark Nottingham">
          <organization></organization>
      <address><email>mnot@mnot.net</email></address>
    </author>
    <date month="October" year="2010"/>
  </front>
  <seriesInfo name="RFC" value="5988"/>
</reference>

<reference anchor="FIPS-180-3">
<front>
<title>Secure Hash Standard (SHS)</title>
<author>
    <organization>National Institute of Standards and Technology (NIST)</organization>
</author>
<date year="2008" month="October"></date>
</front>
<seriesInfo name="FIPS PUB" value="180-3"></seriesInfo>
</reference>

<reference anchor='RFC5751'>
<front>
<title>Secure/Multipurpose Internet Mail Extensions (S/MIME) Version 3.2 Message Specification</title>
<author initials='B.' surname='Ramsdell' fullname='B. Ramsdell'>
<organization /></author>
<author initials='S.' surname='Turner' fullname='S. Turner'>
<organization /></author>
<date year='2010' month='January' />
</front>
<seriesInfo name='RFC' value='5751' />
</reference>

<reference anchor="RFC5854">
  <front>
      <title abbrev="Metalink Download Description Format">The Metalink Download Description Format</title>                
      <author initials="A." surname="Bryan" fullname="Anthony Bryan">
            <organization>Metalinker Project</organization>
            <address>                
              <email>anthonybryan@gmail.com</email>        
              <uri>http://www.metalinker.org</uri>                
            </address>
      </author>
      <author initials="T." surname="Tsujikawa" fullname="Tatsuhiro Tsujikawa">
        <organization>Metalinker Project</organization>
        <address>
          <email>tatsuhiro.t@gmail.com</email>
          <uri>http://aria2.sourceforge.net</uri>
        </address>
      </author>
      <author initials="N." surname="McNab" fullname="Neil McNab">
        <organization>Metalinker Project</organization>
        <address>
          <email>neil@nabber.org</email>
          <uri>http://www.nabber.org</uri>
        </address>
      </author>
      <author initials="P." surname="Poeml" fullname="Peter Poeml">
        <organization>MirrorBrain</organization>
        <address>
          <email>peter@poeml.de</email>
          <uri>http://mirrorbrain.org/~poeml/</uri>
        </address>
      </author>
    <date month="June" year="2010"/>
  </front>
<seriesInfo name="RFC" value="5854"/>
</reference>

<reference anchor="ISO3166-1">
<front>
<title>ISO 3166-1:2006.  Codes for the representation of names of countries and their subdivisions -- Part 1: Country codes</title>
<author>
<organization>International Organization for Standardization</organization>
</author>
<date month="November" year="2006"/></front>
</reference>

          </references>

      <references title="Informative References">

<reference anchor='RFC4732'>
<front>
<title>Internet Denial-of-Service Considerations</title>
<author initials='M.' surname='Handley' fullname='M. Handley' role="editor">
<organization /></author>
<author initials='E.' surname='Rescorla' fullname='E. Rescorla' role="editor">
<organization /></author>
<author>
<organization>IAB</organization></author>
<date year='2006' month='December' />
<abstract>
<t>This document provides an overview of possible avenues for denial-of-service (DoS) attack on Internet systems.  The aim is to encourage protocol designers and network engineers towards designs that are more robust.  We discuss partial solutions that reduce the effectiveness of attacks, and how some solutions might inadvertently open up alternative vulnerabilities.  This memo provides information for the Internet community.</t></abstract></front>

<seriesInfo name='RFC' value='4732' />
<format type='TXT' octets='91844' target='http://www.rfc-editor.org/rfc/rfc4732.txt' />
</reference>
          

<reference anchor='RFC5843'>

<front>
<title>Additional Hash Algorithms for HTTP Instance Digests</title>
<author initials='A.' surname='Bryan' fullname='A. Bryan'>
<organization /></author>
<date year='2010' month='April' />
<abstract>
<t>The IANA registry named "Hypertext Transfer Protocol (HTTP) Digest Algorithm Values" defines values for digest algorithms used by Instance Digests in HTTP.  Instance Digests in HTTP provide a digest, also known as a checksum or hash, of an entire representation of the current state of a resource.  This document adds new values to the registry and updates previous values.  This document is not an Internet Standards Track specification; it is published for informational purposes.</t></abstract></front>

<seriesInfo name='RFC' value='5843' />
<format type='TXT' octets='6731' target='http://www.rfc-editor.org/rfc/rfc5843.txt' />
</reference>


      </references>          


            <section title="Acknowledgements and Contributors">
                  <t>Thanks to the Metalink community, Alexey Melnikov, Julian Reschke, Mark Nottingham, Daniel Stenberg, Matt Domsch, Micah Cowan, David Morris, Yves Lafon, Juergen Schoenwaelder, Ben Campbell, Lars Eggert, Sean Turner, Robert Sparks, and the HTTPBIS Working Group.</t>
                  
          <t>Thanks to Alan Ford and Mark Handley for spurring us on to publish this document.</t>
                  
                  <t>This document is dedicated to Zimmy Bryan, Juanita Anthony, and Janie Burnett.</t>

            </section> 

          </back> 
        </rfc>