[3.13] gh-138907: Support RFC 9309 in robotparser (GH-138908) (GH-149376)

* empty lines are always ignored instead of separating groups
* the "user-agent" line after a rule starts a new group
* groups matching the same user agent are now merged
* the rule with the longest match wins instead of the first matching rule
* in case of equal matches, the “Allow” rule wins over “Disallow”
* special characters “$” and “*” are now supported in rules
* prefer full match for user agent

(cherry picked from commit bc285e5)

Author: serhiy-storchaka

Doc/library/urllib.robotparser.rst

@@ -20,7 +20,7 @@
 This module provides a single class, :class:`RobotFileParser`, which answers
 questions about whether or not a particular user agent can fetch a URL on the
 website that published the :file:`robots.txt` file.  For more details on the
-structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html.
+structure of :file:`robots.txt` files, see :rfc:`9309`.
 
 
 .. class:: RobotFileParser(url='')