Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
robots.txt Manager Details »
robots.txt Manager
Mod Version: 1.00, by MUG (Member) MUG is offline
Developer Last Online: Mar 2010 I like it Show Printable Version Email this Page

This modification is in the archives.
vB Version: 2.2.x Rating: (1 vote - 5.00 average) Installs: 15
Released: 08 Feb 2003 Last Update: Never Downloads: 25
Not Supported Is in Beta Stage  

This script allows you to easily create a dynamically generated robots.txt file, based on specified rules.

If you use this hack, please click 'Install'

Screenshots will be attached...

Download Now

Only licensed members can download files, Click Here for more information.

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
  #16  
Old 09 Feb 2003, 04:09
SphereX
Guest
 
very nice!


***installs
Reply With Quote
  #17  
Old 09 Feb 2003, 11:51
djr's Avatar
djr djr is offline
 
Join Date: Nov 2001
Real name: Jean-Paul
Hi MUG,

Can you add another column 'Owner' and 'Origin' (or whatever you might want to call it) where we can add the owner and origin of the spider?

For example:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Not every spider describes itself fully. e.g. Mercator-2.0 is one of Altavista's robots, but there's no link to Altavista whatsoever.

Thanks,
- djr
__________________
- highly ill, but always intelligent -


- User Age in CP (together with the_sisko)
Reply With Quote
  #18  
Old 09 Feb 2003, 11:54
djr's Avatar
djr djr is offline
 
Join Date: Nov 2001
Real name: Jean-Paul
I found some good overviews of spiders here and here. If anyone has more of these lists, please add them to this thread.

Thanks,
- djr
Reply With Quote
  #19  
Old 09 Feb 2003, 12:24
MUG MUG is offline
 
Join Date: Apr 2002
Originally posted by djr
Hi MUG,

Can you add another column 'Owner' and 'Origin' (or whatever you might want to call it) where we can add the owner and origin of the spider?

For example:

Block Disabled:      (Update License Status)  
Suspended or Unlicensed Members Cannot View Code.

Not every spider describes itself fully. e.g. Mercator-2.0 is one of Altavista's robots, but there's no link to Altavista whatsoever.

Thanks,
- djr
Ooh, thanks. I was wondering what Mercator-2.0 was. aranoid:

I'll add a description field, but there's not enough room for it to show on the main page so you'll have to click edit to view it.
Reply With Quote
  #20  
Old 09 Feb 2003, 12:40
MUG MUG is offline
 
Join Date: Apr 2002
Version 1.0 final released. irate:
Attached Files
File Type: zip robots-1.0r2.zip (6.3 KB, 42 views)

Last edited by MUG; 09 Feb 2003 at 13:11.
Reply With Quote
  #21  
Old 09 Feb 2003, 13:26
MUG MUG is offline
 
Join Date: Apr 2002
Can this thread be moved to the Full Releases forum?
Reply With Quote
  #22  
Old 09 Feb 2003, 18:31
Velocd's Avatar
Velocd Velocd is offline
 
Join Date: Mar 2002
Real name: Mike
I have a slight problem with googlebots, and that is they storm my forum by huge numbers. Currently, for example, I have 7 googlebots crawling my forum. That seems purely excessive to me, and I would like to somehow limit the amount of googlebots to maybe 2.

What is the command line for robots.txt to do this? Or maybe there is some other alternate method.

Thanks
Reply With Quote
  #23  
Old 09 Feb 2003, 18:42
MUG MUG is offline
 
Join Date: Apr 2002
Originally posted by Velocd
I have a slight problem with googlebots, and that is they storm my forum by huge numbers. Currently, for example, I have 7 googlebots crawling my forum. That seems purely excessive to me, and I would like to somehow limit the amount of googlebots to maybe 2.

What is the command line for robots.txt to do this? Or maybe there is some other alternate method.

Thanks
Honestly, I don't think that is possible with robots.txt. If you created something that would dynamically insert text into a robots.txt file based on the number of Googlebots spidering your site, Google might "take the hint" and never come back. :ermm:

Last edited by MUG; 09 Feb 2003 at 18:59.
Reply With Quote
  #24  
Old 10 Feb 2003, 03:43
Velocd's Avatar
Velocd Velocd is offline
 
Join Date: Mar 2002
Real name: Mike
Drat.. :ermm:

Wish it were possible somehow, oh well. My current bandwidth is being consumed quicly by these googlebots, so I guess I'll simply have to restrict them from the threads.
Reply With Quote
  #25  
Old 10 Feb 2003, 13:40
Automated Automated is offline
 
Join Date: Sep 2002
Originally posted by Velocd
Drat.. :ermm:

Wish it were possible somehow, oh well. My current bandwidth is being consumed quicly by these googlebots, so I guess I'll simply have to restrict them from the threads.
restricting them from the threads whats the point of getting spidered then ?
Reply With Quote
  #26  
Old 11 Feb 2003, 22:04
djr's Avatar
djr djr is offline
 
Join Date: Nov 2001
Real name: Jean-Paul
We have two different domains, but only one MySQL-database. Is it possible to place the robots.php on both the domains (and thus using the same tables)?

- djr
Reply With Quote
  #27  
Old 13 Feb 2003, 10:46
djr's Avatar
djr djr is offline
 
Join Date: Nov 2001
Real name: Jean-Paul
Already found it. Just rename the robots_log table to robots_log_domain1 and create another one with _domain2 and update changes in robots.php.

- djr
Reply With Quote
  #28  
Old 16 Feb 2003, 16:48
mheinemann's Avatar
mheinemann mheinemann is offline
 
Join Date: May 2002
Real name: Mike
Installed, works great!
Reply With Quote
  #29  
Old 17 Feb 2003, 00:54
MUG MUG is offline
 
Join Date: Apr 2002
Glad that you like it.

Any suggestions?

Last edited by MUG; 17 Feb 2003 at 00:59.
Reply With Quote
  #30  
Old 17 Feb 2003, 14:19
mheinemann's Avatar
mheinemann mheinemann is offline
 
Join Date: May 2002
Real name: Mike
The only suggestion I can think of is being able to import your current robots.txt

I had disallowed "turnitin" and would like to be able to still block them.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


New To Site? Need Help?

All times are GMT. The time now is 23:29.

Layout Options | Width: Wide Color: