Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
dzhiurgis
on July 4, 2024
|
parent
|
context
|
favorite
| on:
Reddit has updated its robots.txt to block all web...
Google sometimes ignore it when it makes sense (ie big bank accidentally adds login page to ignore) or to check for spam activity (in which case google doesn't use their bot user agent)
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: