Here below there is a Schema of a Canonical Robots.txt file good for SEO.
The Most Important Detail to take into Account if you Install the Robotstxt Module is: Substitute the Default Content with This Below
It Generate a Default Faulty Robots.txt that Exclude the Entire site Directory and can Preclude your Site from Progressing in Ranking…
Here below there is the Default Block to Insert Into the Robots.txt file.
User-agent: *
Crawl-delay: 10
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
Disallow: /sites/all/modules/
Disallow: /sites/all/themes/
Disallow: /sites/default/modules/
Disallow: /sites/default/files/color/
Disallow: /sites/default/files/css/
Disallow: /sites/default/files/fonts/
Disallow: /sites/default/files/js/
Disallow: /sites/default/files/languages/
Disallow: /sites/default/files/webform/
# Files
Disallow: /sites/default/settings.php
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /contact/
Disallow: /logout/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=contact/
Disallow: /?q=logout/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
# Boost
Disallow: /boost_stats.php
# Custom
Disallow: /client/*
Disallow: /search/*
Disallow: /category/*
Disallow: /users/*
Disallow: /*?field_*
Drupal 6 How to Export/Import Article/Nodes:
Drupal Tutorials on Export/Import