Is the database strictly opt-in, where only sites submitted by their creators through the form are indexed in the database?
Are any sites scraped and indexed without asking the site creator for their consent to be included in this search tool? I ask because I have a web space with a /now on a domain that has many other personal sites with /now pages but I do not want my pages to be indexed.
Edit to add: came across this post on hacker news
we built a simple site that indexes 7k+ personal sites [0]
[0] gathered from: 1) https://nownownow.com/ and similar sites 2) checking all HN posts since 2020 with more than 100 upvotes
which similar sites were scraped? which HN posts survived the 100+ up votes since 2020 filter?
Is the database strictly opt-in, where only sites submitted by their creators through the form are indexed in the database?
Are any sites scraped and indexed without asking the site creator for their consent to be included in this search tool? I ask because I have a web space with a /now on a domain that has many other personal sites with /now pages but I do not want my pages to be indexed.
Edit to add: came across this post on hacker news
which similar sites were scraped? which HN posts survived the 100+ up votes since 2020 filter?