Forum Home
Press F1
 
Thread ID: 72120 2006-08-31 00:26:00 Googlebot taking up 1.8Gb Bw and other searchbot queries Morgenmuffel (187) Press F1
Post ID Timestamp Content User
481542 2006-08-31 00:26:00 Hi all

Is this normal?

Name Googlebot
Hits 43077+105
Bandwidth 1.74 GB

The site uses about 6Gb of traffic a month and has about 3000 odd products, but this does seem excessive

also is there a way to get searchbots to spider a phpbb?
is it possible?
is it worth it?

Same as above for coppermine photogallery

Cheers 		Morgenmuffel (187)
481543 2006-08-31 00:40:00 Is this normal?

Name Googlebot
Hits 43077+105
Bandwidth 1 . 74 GB

The site uses about 6Gb of traffic a month and has about 3000 odd products, but this does seem excessive

imo I don’t think that’s too bad, but that depend on how big your site is .

At lest I think your site will get a good listing out of it :)


also is there a way to get searchbots to spider a phpbb?
is it possible?

Yes it is possible; I will find a hack that will allow search engines to crawl phpbb .



Here is the hack I use to allow search engines to crawl my phpbb forums


#
#-----[ OPEN ]------------------------------------------
#

includes/sessions . php

#
#-----[ FIND ]------------------------------------------
#

$SID = 'sid=' . $session_id;

#
#-----[ REPLACE WITH ]------------------------------------------
#

if ( $userdata['session_user_id'] != ANONYMOUS ){
$SID = 'sid=' . $session_id;
} else {
$SID = '';
}

#
#-----[ SAVE/CLOSE ALL FILES ]------------------------------------------
#
# EoM




is it worth it?

Yes, but just a world of warning make sure you are using the current version of phpbb (which is 2 . 0 . 21 as of the date of this post)


Same as above for coppermine photogallery

Have never used coppermine photogallery, so I cant say if you need to do any thing to get search engines to craw it . 		stu161204 (123)
481544 2006-08-31 01:15:00 If you want to store large files on your site or don't want a specific part to be spidered then you should create a file called "robots.txt" in the base directory of your site and put:

User-agent: *
Disallow: /bigfiles/

This will instruct search engines to ignore a particular part of the site. Most will pay attention. 		TGoddard (7263)
481545 2006-08-31 01:51:00 Thanks all

I do have a robots.txt file,
(although i have just updated it, cheers)

as for the phpBB code, where exactly do I add it?
I have version 2.0.20, and don't have time to upgrade as yet forum is quite active so upgrades are done early morning 3am-ish and I am too busy at the moment for a late night
so i'll have to get to that later on

Cheers 		Morgenmuffel (187)
481546 2006-08-31 02:03:00 as for the phpBB code, where exactly do I add it?

PM me your e-mail address & I will PM you mine & e-mail me your includes/sessions.php file & I will add it for you :) 		stu161204 (123)
481547 2006-08-31 05:24:00 The more googlebots hit the better :)

Be careful with the hack. I don't know about that one, but one of the problems using things like that is the possabiity of getting a "duplicate error" issue. Where the same information is accessed using 2 different urls, because one has been changed. If it works OK over a long term then it must be safe however.

I make a google sitemap and upload to my google sitemap account on a regular basis, that indexes my site pretty thoroughly. (for example for one term yesterday on a fake airline announcement in Australia, my site ranked higher than the actual news site I got the information from). 		netchicken (4843)
481548 2006-08-31 06:09:00 Have another look at your logs, Nigel. If Google use 1.8 GB of your connection, what does Yahoo take? At the moment there are about 120 Yahoo Slurping Spiders, one Google, and one MSN spider on this site. :thumbs: Graham L (2)
481549 2006-08-31 06:13:00 Be careful with the hack. I don't know about that one,

It’s fine :), I have used it on a 3 - 4 different phpbb forums & they work fine.

After I add that hack Google & other search engines, crawl my forums well. 		stu161204 (123)
481550 2006-08-31 06:15:00 Actully 1.8 gig is a heck of a lot of bandwidth.

There are cases of the googlebot attacking the site too enthusiastically, I forget the answer but you can contact google to ask for it to be lessened I think. I'll hunt around for an article. 		netchicken (4843)
481551 2006-08-31 06:18:00 At the moment there are about 120 Yahoo Slurping Spiders, one Google, and one MSN spider on this site. :thumbs:

& dont forget 2 AskJeeves Spiders :thumbs: 		stu161204 (123)
1 2