| Forum Home | ||||
| Press F1 | ||||
| Thread ID: 34561 | 2003-06-17 06:11:00 | Spiders from HELL | james (466) | Press F1 |
| Post ID | Timestamp | Content | User | ||
| 153057 | 2003-06-17 06:11:00 | Sorry for the brown-outs recently but, as usual, it was the fault of badly tuned spiders. I know, I know - the same ol' pressf1 technical excuse. But we've blocked them from access so things should speed up. | james (466) | ||
| 153058 | 2003-06-17 06:17:00 | Good work james :) | stu140103 (137) | ||
| 153059 | 2003-06-17 06:31:00 | It may take a while for the blocking to kick in - spiders don't necessarily check robots.txt each visit. | Biggles (121) | ||
| 153060 | 2003-06-17 07:16:00 | Thanks for letting us know people. It's always good to know why it is that it has been down. | -=JM=- (16) | ||
| 153061 | 2003-06-17 09:19:00 | Is it not possible to block the IP that the bad naughty spiders use???? .Chris |
Chris (3346) | ||
| 153062 | 2003-06-17 09:42:00 | > Is it not possible to block the IP that the bad > naughty spiders use???? Yeath, but different search engine use different bots & It will take a wee while to block them all, & some times search engine get new bots on new IP address etc… |
stu140103 (137) | ||
| 153063 | 2003-06-17 22:08:00 | Thought I had found an entomology forum for a minute. | cicero (1379) | ||
| 153064 | 2003-06-17 22:35:00 | > Yeath, but different search engine use different bots why do you spell Yes as Yeath?? do you lithp?? |
oldskool biatch (3927) | ||
| 153065 | 2003-06-17 22:44:00 | > Yeath, but different search engine use different bots > & It will take a wee while to block them all, & some > times search engine get new bots on new IP address > etc… Surely blocking the IPs of KNOWN naughty bad spiders will lead to a reduction of this problem, in much the same way that blocking known spammer domains, such as AOL MSN etc has halved the spam in my inbox. .Chris |
Chris (3346) | ||
| 153066 | 2003-06-18 05:37:00 | Both the robots.txt has been changed and the specific "rogue" spider has been blocked at our firewall. | james (466) | ||
| 1 2 | |||||