Drupal 6 Default Robots.txt for SEO

March 31st, 2012 by Regaz
Here below there is a Schema of a Canonical Robots.txt file good for SEO.

The Most Important Detail to take into Account if you Install the Robotstxt Module is: Substitute the Default Content with This Below

It Generate a Default Faulty Robots.txt that Exclude the Entire site Directory and can Preclude your Site from Progressing in Ranking…

Here below there is the Default Block to Insert Into the Robots.txt file.

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /sites/all/modules/
Disallow: /sites/all/themes/
Disallow: /sites/default/modules/
Disallow: /sites/default/files/color/
Disallow: /sites/default/files/css/
Disallow: /sites/default/files/fonts/
Disallow: /sites/default/files/js/
Disallow: /sites/default/files/languages/
Disallow: /sites/default/files/webform/

# Files
Disallow: /sites/default/settings.php
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /contact/
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=contact/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/

# Boost
Disallow: /boost_stats.php

# Custom
Disallow: /client/*
Disallow: /search/*
Disallow: /category/*
Disallow: /users/*
Disallow: /*?field_*
Drupal 6 How-to Export/Import Article/Nodes:
Drupal Tutorials on Export/Import
(Visited 1 times, 1 visits today)
Share on Tumblr

Tags: , , , ,