Web Design & SEO Company, SEO Service, SEO Web Design, Internet Marketing, Software Development

Admin

Author's details

Date registered: April 13, 2017

Latest posts

  1. Hide Your Robots.txt File From Visitors — August 5, 2017
  2. Steps to Performing a Competitive Analysis — August 5, 2017
  3. Best URL Shorteners to Shorten Long Links — August 4, 2017
  4. Social Media Marketing Tips for Small Business — April 14, 2017

Author's posts listings

Aug 05

Hide Your Robots.txt File From Visitors

Hide your robots.txt file from visitors and show it only for validated robots

How to hide robots.txt file? I will guide.through.

Step-1

the first step you need to add these lines to your .htaccess file. If you don’t have, create one and upload it to the root domain folder.Add the following code to .htaccess file;

Code:

RewriteEngine On

RewriteCond %{http_user_agent} !(googlebot|Msnbot|Slurp) [NC]

RewriteRule ^robots\.txt$ http://internetinfotech.com/  [R,NE,L]

AddHandler application/x-httpd-php .txt

Replace http://internetinfotech.com with your domain name.

 

Step-2

Open a text editor or your favorite web editor application and insert the code below into a new file save as reversedns.php and upload it to your root domain folder.

Code:

<?php
$ua = $_SERVER[‘HTTP_USER_AGENT’];
if(stristr($ua, ‘msnbot’) || stristr($ua, ‘Googlebot’) || stristr($ua, ‘Yahoo Slurp’)){
$ip = $_SERVER[‘REMOTE_ADDR’];
$hostname = gethostbyaddr($ip);
if(!preg_match(“/\.googlebot\.com$/”, $hostname) &&!preg_match(“/search\.live\.com$/”, $hostname) &&!preg_match(“/crawl\.yahoo\.net$/”, $hostname)) {
$block = TRUE;
$URL=”/”;
header (“Location: $URL”);
exit;
} else {
$real_ip = gethostbyname($hostname);
if($ip!= $real_ip){
$block = TRUE;
$URL=”/”;
header (“Location: $URL”);
exit;
} else {
$block = FALSE;
}
}
}
?>

 

Step-3

Open the robots.txt file you would like to protect and insert the code below to the first line.

Code:

<?php include(“reversedns.php”); ?>

 

Step-4

Upload robots.txt file to your domain root folder.