如何阻止过度访问我的网站的机器人？
问题描述：
此机器人不遵守robots.txt中的 nofollow noindex 。
This bot doesn't respect nofollow noindex in robots.txt.
我在机器人中拥有此功能。 txt：
I have this in robots.txt:
User-agent: Msnbot
Disallow: /
User-Agent: Msnbot/2.0b
Disallow: /
到现在为止，这确实很慢，但是现在，这是一个根本不会离开我网站的怪物。
爬行所有WordPress和MyBB 24/7。
Till now it was pretty slow, but now, it is a monster that won't leave my site at all. Crawls all WordPress and MyBB 24/7.
要阻止IP范围，或者我该怎么做才能阻止所有这些内容窃取者？
To block IP ranges or what can I do to stop all of this content stealers?
答
基于按用户代理阻止或空引用，您可以在.htaccess中添加类似的内容
Based on Block by useragent or empty referer you could something like this in your .htaccess
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
SetEnvIfNoCase User-Agent "^Msnbot" ban_agent
Deny from env=ban_agent