fix: robots.txt timeout and multi-agent group parsing (#3 #6)#19

Liohtml merged 1 commit into

masterLiohtml/RUSTScrapling:masterfrom

claude/robots-txt-fixesLiohtml/RUSTScrapling:claude/robots-txt-fixesCopy head branch name to clipboard