| Forum Home | ||||
| Press F1 | ||||
| Thread ID: 17700 | 2002-04-10 00:51:00 | Explanation for 404 errors and other F1 chaos | Guest (0) | Press F1 |
| Post ID | Timestamp | Content | User | ||
| 42931 | 2002-04-10 00:51:00 | Hi everyone, I'm James McNickel and I handle the development and administrative maintenance of PressF1. There have been a lot of 404 errors and general slow downs occurring on F1 all of which have been coursed by GoogleBot - the web spider for Google's search engine. It arrives at indiscriminate times but hit us especially hard last night at around 10.00 p.m. It requests too many pages in too shorter time for our server to handle and as a result brown outs of both the server's cpu and bandwidth occur. Similar problems with spiders occurred last year that we managed to stop either through the robots.txt file or by contacting the search engines directly. We had thought we could handle at least one spider so we had let Google back in. Turns out we can't, and the mistake has been rectified as of this morning. Only problem is I can't say exactly when the spider will re-read our robots.txt file and know that it's not allowed back in. Although, I'd estimate at the most we've got a couple of days before it's gone for good. I wasn't aware that it was hitting us this hard until Robo pointed it out to me and I apologise for any resultant lack of service. Hopefully things will be back to normal soon. James |
Guest (0) | ||
| 42932 | 2002-04-10 01:06:00 | Thanks, James (he's one of the gnomes I was talking about, doesn't really look like a gnome though). Now I think I understand why my wife doesn't like spiders. Should have seen it as a sign when a large one crawled out of the front section of the Herald on to my lap yesterday morning. So we may still experience delays for a couple of days, but we've done what we can. In other words: It wasn't our fault. robo. |
Guest (0) | ||
| 42933 | 2002-04-10 02:20:00 | You need to phone MAF and ask them to do a fly over with their spray can. | Guest (0) | ||
| 42934 | 2002-04-10 07:03:00 | Thanks James, it is always good to get a thorough explanation from someone in the know. JM |
Guest (0) | ||
| 1 | |||||