| Forum Home | ||||
| PC World Chat | ||||
| Thread ID: 72281 | 2006-09-07 03:53:00 | Spiders taking over the (PC) World? | Graham L (2) | PC World Chat |
| Post ID | Timestamp | Content | User | ||
| 483049 | 2006-09-07 03:53:00 | Heres a new trick. A spider replying to a thread: 02:36 PM Yahoo! Slurp Spider Replying to Thread CD-RW: Can't choose fastest speed :badpc: |
Graham L (2) | ||
| 483050 | 2006-09-07 04:29:00 | I mentioned that the other day (www.pressf1.co.nz). They also send threads to their friends and do private messaging. :eek: Funny how the forum is crawling with Yahoo Slurp Spiders these days whereas in the past it was Google bots. The former must have eaten the latter. :rolleyes: |
FoxyMX (5) | ||
| 483051 | 2006-09-07 10:06:00 | got them on my site and they dont seem to have enough legs to press the submit button or it would be flooded with replies | piersdad (5644) | ||
| 483052 | 2006-09-08 09:02:00 | Only thing they do on my site is steal all my bandwidths :( | trinsic (6945) | ||
| 483053 | 2006-09-08 10:37:00 | You can add disallows to your robots . txt, that will stop them wasting bandwidth on dead end links, and mean that the right thread will be indexed in the search engine, and not the reply part, which I have had in the past . I have this in mine Disallow:/post . php? Disallow:/member . php? Disallow:/misc . php? Disallow:/memcp . php? (of course yours will be different to mine) Theoretically threads ending in those strings, or even if they are in the middle of a thread string, it will be ignored by the bot . In practice it doesn't seem that all bots flow the robots . txt . CHINANET bots in particular are rebelious . |
netchicken (4843) | ||
| 483054 | 2006-09-08 12:45:00 | One think we don’t need to worry about is that they cant reply to threads or use PMs or send threads to their friends ;) | stu161204 (123) | ||
| 483055 | 2006-09-08 12:46:00 | Funny how the forum is crawling with Yahoo Slurp Spiders these days whereas in the past it was Google bots. The former must have eaten the latter. :rolleyes: I think you may find is that Google has done a Good index of this forum (& only comes back every so often to update its self) & now its Yahoo Slurp turn to do a good index of this site :) |
stu161204 (123) | ||
| 483056 | 2006-09-08 12:47:00 | Only thing they do on my site is steal all my bandwidths :( How much bandwidth do they use?, How much bandwidth do you have? & How big is your site? |
stu161204 (123) | ||
| 1 | |||||