| Forum Home | ||||
| Press F1 | ||||
| Thread ID: 34561 | 2003-06-17 06:11:00 | Spiders from HELL | james (466) | Press F1 |
| Post ID | Timestamp | Content | User | ||
| 153067 | 2003-06-18 07:51:00 | Now I'm curious and someone has to ask . . . what are the spiders you refer to ? | skidder (3968) | ||
| 153068 | 2003-06-18 08:45:00 | Ever wonder how Google gets it's database of millsions of pages? Spiders (googlebot, in its case). Spiders are automated agents that trawl through and index web pages. Every search engine uses them (they generally are a good thing), but they are also used by less reputable folks for other purposes. There's one called EmailSiphon, for example and its name tells you what it goes looking for on websites and you don't have to be a rocket scientist to work out what is done with all the email addys it harvests. The problem with spiders is that they can swamp a server's ability to serve pages, since they can open multiple sessions on a server with super human speed - in a worst case scenario they can be defacto load testers! Our old PressF1 was brought to its knees by spiders. The current F1 hasn't been effected as badly, but lately it's been intensively hit by web spiders forcing us to take this preventative measure. |
Biggles (121) | ||
| 153069 | 2003-06-18 21:37:00 | Thank you Bruce Buckman et al. I like it when you keep us informed like this. Very helpful and informative of you all. m:Dc | mark c (247) | ||
| 153070 | 2003-06-18 22:38:00 | Im sure they've nicknamed it something like 'Pidgeon Technology' from google... they swamp your server with a flock of birds ;-) | Chilling_Silently (228) | ||
| 153071 | 2003-06-19 22:53:00 | Thanx Bruce | skidder (3968) | ||
| 153072 | 2003-06-25 00:33:00 | Well thats great that its up and running again! The pages still load a bit slow, any idea why? Also can we drop/unstick this topic from the main page? Most people have their settings set to 'display first 10 topics' and most likely don't visit the second page when they are in a hurry. This means that only 9 help threads are showing on the main page?? So every time someone replys, 2 threads get dropped off the main page, insead of the normal 1 thread. >? >? |
PoWa (203) | ||
| 153073 | 2003-06-25 00:52:00 | EDIT> 'display first 15 topics' This means that only 14 help threads are showing on the main page. |
PoWa (203) | ||
| 1 2 | |||||