Defining duplicate content

A search engine considers every unique URL to be the location of a single web page. It learns about such URLs by following links. Duplicate content occurs when a search engine follows more than one unique URLs that take it to a page of predominantly similar content – these pages do not need to be identical, just very similar (a defined ratio is something of a holy grail for optimisers and not clearly established).

Why is this important? A search engine will not display more than two pages from a website for a particular search. It makes an automated decision of which pages are the authoritative or originating version of that content, tucking the rest away at the end of the results: the Supplementary Index.

The damage to SEO comes from a number of factors, the two main causes being:

1. The page you most want to surface may become inadvertently pushed into the Supplementary Index, thus giving searchers an inferior, poorly converting page to visit.
2. A major factor in search result visibility is the number (and quality) of links pointing to a page, but multiple URLs to a page frequently dilutes the combined effectiveness of back-links as the links point to different unique URLs. 

From my direct experience with sites over the past year in particular, duplicate content pages result in a cumulative deterioration of the entire site’s visibility in SERPs (Search Engine Result Pages). 

If you have a site that is failing to rank well for even the least competitive search terms, such duplicate content is likely to be a contributing factor, and you will struggle to establish a firm footing until this problem is resolved.

Spotting the damage

Discovering duplicate content is relatively straightforward. Perform a search for your website on Google, like this:

site:yourdomain.com keyword

where yourdomain.com is your website’s domain, and keyword any terms for which you are trying to be visible. Run down to the very end of the results and if you see a message like this:

you have duplicate content!

The cause

Content Management

If you have a duplicate content issue, then I will bet your website runs on some form of content management. Blog applications are notoriously prone to these problems, with supplementary listing pages such as category summaries, tag summaries, archive lists, and search results, all in danger of generating pages with very similar content.

In addition, some content systems do not chose a single means of generating a URL for a item of content, linking to pages and posts in slightly different ways from different parts of the system. They may also fail to redirect all URLs to the canonical URL for that page or post.

Identical meta data

In addition, it is very common to come across sites with many – sometimes even all – their pages with identical titles and META descriptions . This is particularly prevalent once again with blogs, and also in the small business space where a company feels it must apply its company name and corporate blurb at the top of every page, thus not accurately reflecting that page’s actual content.

Page titles are very important in establishing the context of a page, and must be unique in order for the search engine to properly understand the page content.

Too little unique textual content

Pages with little unique text content can become regarded as duplicative because the majority of the content there is similar to everywhere else on the site. In these sparse content examples, the site-wide navigation, footer information, and other generic text, can form the majority of the textual content.

Search, archive and summary pages

As mentioned above, pages that summarise and list snippets of other content can easily appear to a search engine to be very similar but have unique URLs.

www and non-www domains

This one often surprises web developers, but www.yoursite.com is, to a search engine, a different website to yoursite.com. This means that if all your content is reachable via both those versions, your entire site is seen as being a duplicate!

It does not matter which you choose, but have one of those URL versions permanently redirected to the other.

Repairing the damage

Now that I have covered most of the indicators and causes, how about a means of fixing the problem? Watch out in a few days for specific techniques to repair and remove duplicated content within search engines.

What follows is a tirade of growing intensity while the client – accompanied by gibbon-like arm waving to demonstrate his interlinking, money-earning online website concept – while the the developer sits and struggles to grasp what he’s going on about – while trying to ensure his child is suitably lubed up with caffeine, I should add.

Over half an hour or so, the end of which was signified by the client abruptly switching his attention to his Blackberry to the exclusion of everything else, I hear much of the idea the client presents – sells to the developer (off goes that alarm bell again). What I did not hear, even once, was a proposition on how this wonderful concept, which was “just like so-and-so’s idea only better and he made millions but we can do it with hardly lifting a finger once you build the site”, was actually going to become a real, viable, profitable business.

When the client has to sell the concept to a developer, he is wanting more than simply someone to design-build a product. What he wants, is someone to work for equity, for a promise, or from a percentage of some future profit – a profit which, you will remember, has had not one syllable of substantiation.

There’s a subtle difference between a client briefing you on a project and the client persuading you to take on the project, and it is a difference the freelancer must learn to spot.

Ask to see the business plan

The moment you spot someone selling you a concept, and particularly once they have injected the suggestion about working for equity or a cut of future earnings, there is only one course of action: ask to see the business plan.

Then sit back and watch across the table at the avoidance, fidgeting, and all manner of excuses about it not being ready, incomplete, or “haven’t really worked out the details yet”. The best of all is “we don’t need one there’s no investment and it’s easy money”. Any kind of clear avoidance tactic to not offer a business plan – and one is assuming here there is already an NDA in place, which certainly should be the case prior to a meeting if you are going to discuss a new project concept.

It is at this point you discover why the first developer “never came back”, and at this point you finish up your coffee and politely end the meeting.

Show me the money

Providing you have the portfolio to demonstrate your ability to deliver, lay out a charging agreement that prevents you from undertaking any work until an initial percentage is paid. In cases such as described above, you can be fairly sure you will hear nothing more of it – and you should be grateful of that fact.

When you are politely reminded that the concept requires no investment, feel free to remind the client that while they sit back and watch you work your magic, you are not earning enough money to pay the rent and feed mochas to your hyperactive child.

Saturated market

The space is saturated with hungry developers eeking out a living on increasingly devalued skills, and equally flooded with preying, pseudo-preneur clients keen to exploit a newcomer to the business with the promise of something sweet for the portfolio. Though I have to say some are not so deliberately contriving and genuinely believe you are able to sweat for a promise while they sit back and watch. I’ve been caught by a few myself over the years but have thankfully walked away from many more.

Watch out for the predators!