Blocking Bots with Nginx by PanzersGhost in technology

[–]PanzersGhost[S] 1 insightful - 1 fun1 insightful - 0 fun2 insightful - 1 fun -  (0 children)

"But as I understand it, the one shortcoming of robots.txt is that it only works if visiting bots actually honor your robots.txt file. (Google has a good intro on this, if you’re interested.) That’s why I’ve opted to use my site’s .htaccess file to block these bots. As a friend put it recently, robots.txt is a bit like asking bots to not visit my site; with .htaccess, you’re not asking. (If you’re wondering if robots that ignore robots.txt would perhaps lie about their user agent, you’re right to do so. Relying on user agent strings — which are basically swamps filled with lies — is incredibly fraught in any context, including this one. That’s why I consider this approach to be marginally better than robots.txt, not a perfect solution.)"

https://ethanmarcotte.com/wrote/blockin-bots/

New Here by johnnydunman in Internet

[–]PanzersGhost 2 insightful - 2 fun2 insightful - 1 fun3 insightful - 2 fun -  (0 children)

Paywall Reader - Read without paywalls for free by PanzersGhost in Internet

[–]PanzersGhost[S] 3 insightful - 1 fun3 insightful - 0 fun4 insightful - 1 fun -  (0 children)

You are welcome.