| Forum Home | ||||
| Press F1 | ||||
| Thread ID: 19835 | 2002-05-23 22:50:00 | A note about posting problems | Guest (0) | Press F1 |
| Post ID | Timestamp | Content | User | ||
| 50409 | 2002-05-23 22:50:00 | Just a note to clarify some confusion about what is going on with postings. Firstly, PressF1 is currently running on our main web server, which means it is sharing resources with all the other main parts of the IDG sites - IDGNet, PC World, Unlimited etc. All the sites running on the main server are Lotus Domino databases and the pages are generated dynamically each time one is requested. This can obviously put lots of load on a server, so what we do is cache certain pages. The home pages, including the home page of Pressf1, are cached for 15 minutes in most cases while many other pages (news stories in IDGNet, for example, BUT not PressF1 postings) are cached for 2 hours. Any of you who've been visiting the IDG sites for a while will remember how slow it could get accessing some pages. Caching them has essentially solved this problem, especially in peak times. What this means for PresF1 is that new posts don't show up on the home page until the cache expires and is renewed by someone requesting the page. So the delay is a maximum of 15 minutes (the cache period). However, we can't cache the postings since that would prevent you seeing your new post. Originally we just left them uncached, but due to the nature of PressF1 as a forum and the amount of traffic it gets (not all of it human, but more on that later) this meant that the pages could be very slow to load. Again, those of you who have been here a while will remember this. So what Jim Mcnickel did was use script to generate the threads as static HTML, to remove the burden of dynamic generation. This has worked fine for the most part, since PressF1 is way faster than it used to be. There is, however, a problem - spiders. Search Engine spiders hit the server on a regular basis and even though we tried disallowing them from visiting certain sections (PressF1) the bad ones ignore things like robots.txt and come in anyhow. What this means is that they put a heavy load on the server since they start trawling rapidly though the site, hitting pages that aren't cached. Due to an annoying quirk of the current version of Domino, the HTTP headers - which otherwise could contain date information that would allow us to tell spiders that certain pages are not new and have been indexed before - gets reset with fresh date info each time the page is generated. So spiders essentially think every page is new since they last visited and try to index the lot. Aaagggh. What this means for PressF1 is that the CPU utilisation on the server gets very high and when this happens the HTML generation of pages (when you post a new thread or reply to one) sometimes times out as a result and doesn't create or recreate the thread properly. That's when you see a blank page or and error page. You may have noticed this problem seems to occur in cycles, usually for a few days - that's when spiders are hitting us. The Fix: The new forum we'll roll out soon does not run on the Domino server but on a Linux box and uses JSP pages. Apart from the improved features it will offer we hope that this switch alone should mean it won't suffer from these spider-related problems. Bruce Buckman Web Dev Manager IDG Communicationd Ltd |
Guest (0) | ||
| 50410 | 2002-05-23 23:13:00 | Bruce, I'm looking forward to the new system, as I'm sure many others are. Any ideas on a due date? I thought Robo had said something about the 17th, but that was a week ago now... maybe I read his post wrong. Mike. |
Guest (0) | ||
| 50411 | 2002-05-23 23:22:00 | Thanks for the info, Bruce. The problem faced when one is trying to make a post is that the blue progress bar can take forever and a day to creep along. So that is maybe why people click away. The impression is gained that the posting wont be made until that bar has finished it's thing and the main page comes back. However it would seem in most cases the posting is made before the progress bar finishes. Roll on the new sytem! |
Guest (0) | ||
| 50412 | 2002-05-23 23:23:00 | Keep up the good work. I can tolerate a few imperfections as it is not a paying site. Will look forward to the new site. |
Guest (0) | ||
| 50413 | 2002-05-23 23:55:00 | Most of the work on the new system is complete. The main thing left to do is the registration system. Guests will be able to read posts but to post you'll need to be a registered user. So once you're registered you won't have anyone impersonating you :-) It will also let us boot out any idiots like Mr Cyph0r. Can't stop them re-registering, of course but hopefully getting locked out all the time will bore them in the end. Robo will also be able to access info on which users have been doing the most posts which will help in in handing out prizes. The only problem is that like most registered forums this system does use cookies to set your log-in state, so anybody who has diabaled all cookies will have to enbale them for PressF1 if they want to post. The cookies are there purely for this purpose - no evil tracking of Interent usage or anything like that, we promise. |
Guest (0) | ||
| 50414 | 2002-05-24 02:30:00 | Thanks for the info Bruce. It seems as if the post has gone through as soon as the 'opening page' message in the status bar appears. JM |
Guest (0) | ||
| 50415 | 2002-05-24 22:37:00 | Great update, Bruce, good on you. I too am a very appreciattive user and thank you and all the others for maintaining such a great site. Only discovered couple of weeks ago and now it's my home page. | Guest (0) | ||
| 1 | |||||