Conversation
| Disallow: /_image? | ||
| Disallow: /Benutzer/ | ||
| Disallow: /users/ | ||
| Disallow: /*?flavour=mobile |
There was a problem hiding this comment.
Although the URLs do no switch to a mobile version any more, they are still valid (f.e. https://ubuntuusers.de/?flavour=mobile displays the same content like https://ubuntuusers.de/ with HTTP 200). So a possible drawback could be duplicate URLs in the search index, because links still use this variant or are still in the google index etc. Thus, i would avoid this removal. Any other opinions?
There was a problem hiding this comment.
Yes, that’s probably right. However fixed now.
There was a problem hiding this comment.
If the flavour parameter is deprecated, could we instead use redirects? Then search engines and alike would take notice and not crawl them.
There was a problem hiding this comment.
IMO that's hard to do within Inyoka, as this potentially affects every view (and also 'only' old intances). I dont know whether the used proxy has the ability to 'remove' all flavour HTTP-Parameter with a 301.
| Crawl-delay: 10 | ||
| Disallow: /search | ||
| Disallow: /_image? | ||
| Disallow: /Benutzer/ |
There was a problem hiding this comment.
If /Benutzer/ is blocked for bots, shouldn't /user/ be blocked too?
There was a problem hiding this comment.
Anonymous users see at /user/-URLs a login page, so i dont see really a benefit.
There was a problem hiding this comment.
But they see a login page at /Benutzer/ too. So wouldn’t it be consequent also to remove that line and the /users/ one?
There was a problem hiding this comment.
Valid Point. However, to resolve this i would suggest another thing: Return not HTTP 200 on login pages, instead 403 (or 404, if we want to hide the page's existence).
There was a problem hiding this comment.
As the user names and links to it's profiles are known, we don't need to and can hide the pages' existence.
3 participants
This pull request removes no more needed entrys in robots.txt file.