Skip to content

several bots not recognized #84

@FlorianLudwig

Description

@FlorianLudwig

I manually looked through some logs and feed it into httpagendparser and found several wrongly identified user agents, including high profiles google bot:

>>> httpagentparser.detect("Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
{'platform': {'name': 'Android', 'version': '6.0.1'}, 'os': {'name': 'Linux'}, 'bot': False, 'dist': {'name': 'Android', 'version': '6.0.1'}, 'browser': {'name': 'Chrome', 'version': '41.0.2272.96'}}

>>> httpagentparser.detect("Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0")
{'platform': {'name': 'Linux', 'version': None}, 'os': {'name': 'Linux'}, 'bot': False, 'browser': {'name': 'Firefox', 'version': '38.0'}}

>>> httpagentparser.detect("crawler_eb_germany")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)")
{'platform': {'name': None, 'version': None}}

>>> httpagentparser.detect("ZoominfoBot (zoominfobot at zoominfo dot com)")
{'platform': {'name': None, 'version': None}}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions