Google's Double Standard on Search Engine Cloaking

March 3, 2007: Please read the updates at the bottom of this post. It looks like this issue with Webmasterworld's cloaking and Google has finally been addressed.

I've debated about posting on this topic for a long time, but I am having regular frustrating experiences with Google. This problem has been going on for years.

It appears that Google has a double standard when it comes to cloaking content.

Here are Google's Webmaster Guidelines about cloaking:

Make pages for users, not for search engines. Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."

Note that this is the first entry in their section of webmaster guidelines about basic principles. Google also says the following on the same page, under a header called Specific Guidelines:

Don't employ cloaking or sneaky redirects.

It cannot be said more plainly than that. It is clearly against Google's policy to serve different content to search engines than to human visitors.

I frequently search for answers to technical questions in Google. Two sites that I frequently (and accidentally) click on are WebmasterWorld.com and Experts-Exchange.com. When I arrive on their sites, I am not presented with the content that Google has promised; I am given a login screen. This is cloaking.

Webmaster World's Cloaking

WebmasterWorld has been cloaking for a long time. Sometimes the site is cloaked and sometimes it isn't. I haven't figured out the pattern yet. Generally, when I click on a Webmaster World result in Google (which is very common), I get a login screen saying that I needed to register or login. A second visit with the browser's user agent set to Googlebot would show that the full content was being sent to Googlebot (although with the link structure changed).

Here is a sample WebmasterWorld.com result in Google's search engine:

Note that in the above image that Google does not have a cache of the page.

Here is a screenshot of the login screen that is only shown to human visitors:

Note in the above screenshot that it appears that you must pay to view the content. Everything on the page seems to indicate that you must pay to view the content. However, there are just a couple of tiny links on the page that allow you to register for free. Google just plays along with this little game.

WebmasterWorld.com's Cloaking Tricks

Webmaster World blocks search engines' caching of their pages with the following HTML:

<meta name="googlebot" content="noarchive">
<meta name="robots" content="noarchive">

This prevents Google from caching the pages. The only way to tell that they are cloaking is to view the site with the Googlebot user agent. Webmaster World has also threatened to ban any IP where someone views the site with a search engine's user agent:

# Any unauthorized bot running will result in IP's being banned.
# Agent spoofing is considered a bot - if it looks like a bot and not from an SE - it is a bot.

Is that to reduce the chance that people will discover that Webmaster World is cloaking?

Also take a look at their robots.txt file. It disallows all robots:

It is very creative to put a blog in the robots.txt file, but the presence of the disallow rule is a dead give-away that more cloaking is taking place. A quick visit with user-agent set to Googlebot confirms that. Googlebot sees a different robots.txt file.

Experts-Exchange.com's Cloaking

Experts-Exchange.com has many high ranking pages. They seem to be ranking lower than they used to, but I still regularly find myself accidentally clicking on their pages in the Google SERPs.

Here is a sample result in Google from experts-exchange.com — as of today it ranks on the first page of Google for gui ubuntu server:

This is the URL http://www.experts-exchange.com/Operating_Systems/Linux/Q_21975778.html.

Clicking on the link in Google does not lead you to a page with a solution the the problem you are searching for. It just gives you a login screen that says "View Solution", as shown below.

Clicking on the "View Solution" button leads to a page that says that you have to subscribe to view the solution, as shown below:

Because I have been frustrated with Expert's Exchange clogging up the SERPs for years, and it seemed impossible that they could rank so well in Google with such little content on each page, I took a look at the Google cache of their page to see if they are cloaking. As shown below, the page that is served to Googlebot is different than the one that is served to the visitor. The full content is cached by Google at the bottom of the Expert's Exchange pages as shown by the red arrow below:

This is blatant cloaking, and it is extremely frustrating to continually click on these links in the Google SERPs, both on Expert's Exchange and Webmaster World.

SearchEngineWatch.com has a thread about Experts-Exchange's cloaking.

Damaging the Web

This kind of cloaking is detrimental to the Web. What if every site started doing this kind of cloaking? What if every time you clicked on a result in Google you had to register and login to the web site? It is an extremely frustrating problem to click on a search engine result and then have to register or login for the site. Both sites also have misleading login screens that on first glance look like you have to pay to register.

Google, please stop applying a double standard on cloaking.

If you find this problem annoying and frustrating, let Google know about it. Perhaps leave a comment in Matt Cutt's blog on the thread of complaints about Webmaster World's search engine cloaking.

Updates

UPDATE, March 3, 2007: Matt Cutts addressed this issue today. Has Webmasterworld changed and stopped cloaking? I was surprised that Matt Cutts didn't mention my post because I'm fairly sure that Matt Cutts has read this page. This page was also written just several days before Philipp Lenssen's and I had always assumed that Philipp had read this article first. Maybe a coincidence.

Brett Tabke from Webmasterworld posted a comment on Matt Cutts' blog post that said:

I don’t blame anyone for being upset that there ISP was on a required login - I also get miffed when we click through to NY times and are forced to login.

It is nice to know that he understands the issue that search engines visitors are faced with. I hope that the issue is resolved.

Brett Tabke talks about the new system here. It will be interesting to watch how the story unfolds.

Related Post

• Pubcon #1 in Google for "robots.txt"