Register Members List Search Today's Posts Mark Forums Read

Reply
 
Mod Options
robots.txt Manager Details »»
robots.txt Manager
Mod Version: 1.00, by MUG (Member) MUG is offline
Developer Last Online: Mar 2010 I like it Show Printable Version Email this Page

This modification is in the archives.
vB Version: 2.2.x Rating: (1 vote - 5.00 average) Installs: 15
Released: 08 Feb 2003 Last Update: Never Downloads: 25
Not Supported Is in Beta Stage  

This script allows you to easily create a dynamically generated robots.txt file, based on specified rules.

If you use this hack, please click 'Install'

Screenshots will be attached...

Download Now

Only licensed members can download files, Click Here for more information.

Show Your Support

  • To receive notifications regarding updates -> Click to Mark as Installed.
  • This modification may not be copied, reproduced or published elsewhere without author's permission.
Comments
  #2  
Old 08 Feb 2003, 21:00
MUG MUG is offline
 
Join Date: Apr 2002
Control Panel
Attached Images
File Type: jpg screen1.jpg (119.0 KB, 763 views)

Last edited by MUG : 12 Apr 2003 at 16:00.
Reply With Quote
  #3  
Old 08 Feb 2003, 21:01
MUG MUG is offline
 
Join Date: Apr 2002
Editor (All Robots)
Attached Images
File Type: jpg screen2.jpg (54.4 KB, 457 views)

Last edited by MUG : 12 Apr 2003 at 16:01.
Reply With Quote
  #4  
Old 08 Feb 2003, 21:01
MUG MUG is offline
 
Join Date: Apr 2002
Editor (Specific Robot)
Attached Images
File Type: jpg screen3.jpg (42.4 KB, 319 views)

Last edited by MUG : 12 Apr 2003 at 16:01.
Reply With Quote
  #5  
Old 08 Feb 2003, 21:01
MUG MUG is offline
 
Join Date: Apr 2002
Generated File (:banana
Attached Files
File Type: txt robots.txt (1,023 Bytes, 187 views)

Last edited by MUG : 12 Apr 2003 at 16:04.
Reply With Quote
  #6  
Old 08 Feb 2003, 21:08
Neo's Avatar
Neo Neo is offline
 
Join Date: Oct 2001
Real name: Dan
Excellent G.
Reply With Quote
  #7  
Old 08 Feb 2003, 21:31
Dean C's Avatar
Dean C Dean C is offline
 
Join Date: Jan 2002
Real name: Dean Clatworthy
Wow very nice!
__________________
Dean Clatworthy - Web Developer/Designer
Reply With Quote
  #8  
Old 08 Feb 2003, 22:04
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
very good job, tho' I can edit one txt file by hands No offence, again, very good job

May I ask you if you have the list with IPs of some major/all search engines? I kinda need it Thanx a lot and again, a nice hack!
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.
Reply With Quote
  #9  
Old 08 Feb 2003, 22:07
MUG MUG is offline
 
Join Date: Apr 2002
The point of the hack is to make administration easier. It keeps track of robots requesting the robots.txt file, allowing you to ban or restrict a bot without having to dig through the server logs. I wrote the hack today, so the only bots included in the database already are the ones that spidered my site during that time. The list is in the .sql file.
Reply With Quote
  #10  
Old 08 Feb 2003, 22:37
wooolF[RM]'s Avatar
wooolF[RM] wooolF[RM] is offline
 
Join Date: Jan 2002
thanx for the answer
__________________

It's nice to be important but it's more important to be nice.
--------------------------------------------------------------------------------------------------------------------------------------------
Never discuss with an idiot, he'll just pull you down to his own level and beat you with experience!
--------------------------------------------------------------------------------------------------------------------------------------------
Its not the penis that makes the man, its the way he uses it.
Reply With Quote
  #11  
Old 08 Feb 2003, 22:50
djr's Avatar
djr djr is online now
 
Join Date: Nov 2001
Real name: Jean-Paul
Is it suppose to write a new robots.txt file everytime or do the bots see the robots.php file as robots.txt?

If your answer is it's suppose to write a new robots.txt file, it isn't working for me :-(

And: do I still need a robots.txt file?
Reply With Quote
  #12  
Old 08 Feb 2003, 23:06
MUG MUG is offline
 
Join Date: Apr 2002
Originally posted by djr
Is it suppose to write a new robots.txt file everytime or do the bots see the robots.php file as robots.txt?

If your answer is it's suppose to write a new robots.txt file, it isn't working for me :-(

And: do I still need a robots.txt file?
It uses mod_rewrite to send requests to robots.php. You have to create an .htaccess file with the following:

RewriteEngine on
RewriteRule robots.txt /robots.php (note: this is for the old version. read the new install file )

Upload robots.php to the root web directory (usually public_html). Make sure you run robots.sql using phpMyAdmin.

Last edited by MUG : 09 Feb 2003 at 18:23.
Reply With Quote
  #13  
Old 08 Feb 2003, 23:33
djr's Avatar
djr djr is online now
 
Join Date: Nov 2001
Real name: Jean-Paul
I did that already
So the robots are redirected to robots.php which is feeding them a perfectly rendered robots.txt file? Sorry 'bout asking, but I don't want to break my (high) ranking(s).

- djr
Reply With Quote
  #14  
Old 09 Feb 2003, 00:10
MUG MUG is offline
 
Join Date: Apr 2002
Originally posted by djr
I did that already
So the robots are redirected to robots.php which is feeding them a perfectly rendered robots.txt file? Sorry 'bout asking, but I don't want to break my (high) ranking(s).

- djr
Yup. The only difference is the X-Powered-By header generated automatically by PHP.
Reply With Quote
  #15  
Old 09 Feb 2003, 00:16
BigCheeze BigCheeze is offline
 
Join Date: Oct 2002
Thanks! I just installed it. See if I can control those bot's a little more!!
__________________
Got Network? | Need review? | Autism support
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Mod Options

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
Forum Jump


New To Site? Need Help?

All times are GMT. The time now is 10:30.

Layout Options | Width: Wide Color: