| Forum Home | ||||
| PC World Chat | ||||
| Thread ID: 74209 | 2006-11-15 01:37:00 | Yahoo! Slurp Spiders? | SurferJoe46 (51) | PC World Chat |
| Post ID | Timestamp | Content | User | ||
| 499218 | 2006-11-15 01:37:00 | I see that there are a few of those slurpers looking at profiles...is this a concern? | SurferJoe46 (51) | ||
| 499219 | 2006-11-15 02:05:00 | Nah they are just indexing profiles into Yahoo search :) | trinsic (6945) | ||
| 499220 | 2006-11-15 02:07:00 | Never pays to have anything in a public profile you don't want ... well ... public. | Biggles (121) | ||
| 499221 | 2006-11-15 02:19:00 | Yahoo's searching must be very inefficient -- there are still over 170 of them running all the time, looking at everything (though the don't seem to be "sending" private messages any more). Perhaps they are indexing every character on the site: so you can search for all occurrences of, e.g., "a". :D But even if they were doing that, you would think that they might have found everything in the archives by now, and could be like Google, with only one or two spiders keeping up to date. In fact, they would probably get more results quicker if they didn't run so many spiders ... the server isn't parallel, and the swapping between 170 odd client tasks could be using more time than the tasks are getting for useful work. :cool: | Graham L (2) | ||
| 499222 | 2006-11-15 03:18:00 | Yahoo's searching must be very inefficient -- there are still over 170 of them running all the time, looking at everything (though the don't seem to be "sending" private messages any more). Perhaps they are indexing every character on the site: so you can search for all occurrences of, e.g., "a". :D But even if they were doing that, you would think that they might have found everything in the archives by now, and could be like Google, with only one or two spiders keeping up to date. In fact, they would probably get more results quicker if they didn't run so many spiders ... the server isn't parallel, and the swapping between 170 odd client tasks could be using more time than the tasks are getting for useful work. :cool: So...let me get this straight... Y! keeps searching here looking for changes and modifications to the posts and eVerything else..right? I use a sniffer program to watch several URL's too and they report to me when there's changes to the sites I want to watch. I guess it's about the same...no? Can one actually see the slurpers in action watching sites that...say...I subscribe to via my sniffer.....or is it just something that F1 allows us to see because we live here? |
SurferJoe46 (51) | ||
| 499223 | 2006-11-15 03:22:00 | Terrorists. FBI. George W Bush. Al Queda. Osama Bin Laden. September 11. | pine-o-cleen (2955) | ||
| 499224 | 2006-11-15 17:17:00 | Terrorists. FBI. George W Bush. Al Queda. Osama Bin Laden. September 11. Wow...let me c/p those in my Sex, Free Drugs & Rock and Roll post..... |
SurferJoe46 (51) | ||
| 499225 | 2006-11-15 17:54:00 | This site gets indexed fast, and threads here are being indexed within days of being posted. Do you have a robots.txt file? You can prevent the bots from indexing all areas like this that serve no purpose. |
netchicken (4843) | ||
| 499226 | 2006-11-15 19:48:00 | Do you have a robots.txt file? You can prevent the bots from indexing all areas like this that serve no purpose. Hey that's a mighty big value judgement. I think F1 Chat serves a major purpose and I'm glad that all the deep philosophy and intelligent debate found here is making its way into the major search engines. Some of the gems found here deserve to be shared with the world ... |
Biggles (121) | ||
| 499227 | 2006-11-15 22:10:00 | Ahh I meant "like this" as in the profiles. There is not much sence in the profiles being indexed Lets not jump to fast here :) |
netchicken (4843) | ||
| 1 2 3 | |||||