Drupal.org SEO Part 3

This is part 3 of a case study on how Drupal.org could be further optimized for search engine rankings.

Google has indexed over 2,500 pages on the subdomain www2.drupal.org. Here is a screenshot:

Since www2.drupal.org is a duplicate of drupal.org, Google is indexing duplicate content on the site which can hurt rankings. It also puts extra load on the servers because of the extra pages being crawled.

There are two possible solutions:

1. Send 301 redirects from all pages on the www2 subdomain to their corresponding pages on the main domain drupal.org.
2. Alternatively, block all pages on the www2 subdomain from robots with the robots.txt file.

The first option could be implemented with .htaccess. The second option could be implemented by having the URL http://www2.drupal.org/robots.txt serve the following content:

User-agent: *
Disallow: /

That would prevent search engines from crawling and indexing the duplicate content. (The main robots.txt file at http://drupal.org/robots.txt would serve different content—the regular robots.txt file.)