SEO - Fixing  DotNetNuke Terms of Use and Privacy Statement links to remove duplicate content

Being concerned with Search Engine Optimization (a.k.a. SEO), I have been meaning to do this for a while now, but it always seems to fall to lower priority on my list, mostly because I don't believe duplicate content is a major concern in SEO.  I know this topic has been covered before, but I think I have a little different approach, and one that will give you some ways to do real SEO with your page names and content.

I have discussed using the SiteUrls.config for cleaning up the portal alias query string in another post, but this one will take it even further to show you how to really get a handle on one of the problems of having "one size fits all" Friendly Url Rewriter rules.  Even if you are not concerned with Terms and Privacy links this tutorial can help you customize any Url on your site to be exactly what you want.

Are there really duplicate pages?
OK, first we will focus on the terms of use and privacy links since most everyone has these two links on their DotNetNuke portals, and there are some clear advantages to getting these links changed. Lets start by proving the problem is real.

Anytime you see "omitted results" in a search results on Google it means you have been "tagged" as having what looks like duplicate content.  Is that a bad thing?  Probably not for most people.  Unless you are intentionally duplicating content to do "Blackhat" type SEO then you should see no negative results. But hey, why not take the opportunity and remove the worry, while at the same time maybe even adding to your SEO arsenal?

You can get those results for your own site by running a query on Google that looks like the following link:
http://www.google.com/search?q=allinurl:+ctl+privacy+site:www.dotnetnuke.com&hl=en&lr=&as_qdr=all&filter=0

Correcting the problem

As mentioned earlier there's more than one way to correct this problem, but I chose this way because it is a technique that I believe is easy and flexible. As a brief overview, what we are going to do is:

1. Create new pages in DotNetNuke and add a text/html module to display the information.
2. Make the new pages accessible with friendlier Urls.
3. Correct the output of our pages so that we do not expose multiple links to the same content. (optional, but avoids duplicate content)

#1.   ADD NEW PAGE:  The reason we want to create a new page is because it gives us the ability to modify the content and set the page skin just like other content on our site.  Let's be honest, who has even changed the content that is automatically generated for the Terms of Use and Privacy Statements?  By creating new pages and putting a normal html module on it you can easily change the content. Yes I know this removes the the ability to customize it by language, but that is probably not a problem in 90% of sites.

 Add a new page and give it a descriptive name with a good Title and Keywords. 
   TIP:  While you are here, you could do some creative SEO with this new page.

You'll also want to make it visible to all users and you probably do not want it to show in the regular navigation so uncheck "Display in Menu".

Now you can click on the terms of use or privacy statement link you currently have and copy the text from that page, then paste it into a text/html module on this new page.    TIP:  While you are here, you could do some creative SEO with this new module content.

Now make a note of the TabId so that it can be used in the second step.

#2.  UPDATE SiteUrls.Config: This is so that the Page can be accessed without needing the TabId and the ctl=Privacy or ctl=Terms.
   TIP:  This method can be used to create really friendly Urls to any page that you want to be indexed better or to just make the Urls look better in print marketing, etc.

The Rewriter Rule above looks for an incoming Url that has our Friendly Page Name and then rewrites it internally in DNN so that the correct page is displayed.
Note: If you have a DNN install with multiple portals then you will need unique names for your Terms Of Use and Privacy Statement pages that match the tabid to the correct page for the portal.

#3.   CORRECT THE OUTPUT URLS: OK, now we have content, and a way to address the content in a friendly way so that it shows up, so now we just need to make sure that we do not output the duplicate urls causing spiders to index the same page using a different url.

One way to do this would be to go in and remove the [TERMS] and [PRIVACY] Skin Tokens, then replace them with hard links to our nice new page names.  That is a tedious process though, so what we really need is a way to capture the Urls before they go out to the browser and replace (rewrite) them to our new friendly page names.  This is where the Free PageBlaster can really help you out.  It has a global output replacement filter to handle just these types of tasks.

After installing PageBlaster you can update the Snapsis.PageBlaster.config file to include replacement rules for the output of your pages.

The Replacement Rules above will capture the links that are output by the [TERMS] and [PRIVACY] skin tokens and replace them with the friendly names.

That's it!  I'll be happy to answer any questions or listen to feedback in replies to this post.

Hi John

I was not aware of the duplication relevance of "Google omitted results" thanks for the useful information.

I have seen this message when I have Googled for my site. When I repeat the search with expanded omitted results, I am surprised to find several pages from my site with what I believed to be non duplicated and very different content, including, at least 3 pages associated with the home page:

http://www.mysite/com/Default.aspx?
http://www.mysite/com/Home.aspx
Which both take me to the site home page.

http://www.mysite/com/Terms.aspx? (using the page title from the home page) but this brings me to the old terms page not the new custom SEO created Terms page.

Also, when I click on the Login, Register, Terms, Privacy pages the Breadcrumb displays:
You are Here: Home (instead of the relevant associated page) Plus the page title for the above is the same as the home page.

Do you think this might have some bearing on the Google page duplication result and if so how might this be rectified?


Regards

Lynn

Hi Lynn,
One thing that will cause Google to tag pages as similar is the Title, Description, and Keywords being the same.  This happens on those pages because when you click on one of those links it still uses the home page to load in the appropriate control.

The only way to get around it is with a techniquie much like this one.  Under Admin > Site settings you can assign different unique pages to some of these controls and it's not a bad idea to do so.

It's not a something you should be overly concerned about though.  Duplicate page penalties are way over-hyped in my opinion.  If you have the choice between removing these duplicate pages and creating a brand new page of helpful and key word rich content then I would opt for the later.

There is one other thing you can do.  Place a robots.txt file in the root of your site that tells the search engines not to index pages that have "ctl" on the querystring.  You can see an example by looking at mine: http://www.snapsis.com/robots.txt

The Terms.aspx page is probably not working because you have another rewriter that is making the text in the ouput Url different than what this tutorial looks for.  We are looking for "ctl" in the querystring, but your url is already optimized so it doesn't find it.  In other words, you don't really need to implement this unless you want to have that content on a real page.  If so, just change the SiteUrls.config rule to look for the Url that your page is using :

LookFor : .*Privacy\.aspx

SendTo :  ~/Default.aspx?tabid=nnn <-- where nnn is the tabid of your new privacy page.

Hi John,

I've got the latest version of pageblaster and have made sure all my settings are correct in my web.config and all the proper files are in place.  I know it is working because pages are being cached.  I have friendly URLS checked for my portal and have followed this tutorial the best I could but I still have had no success with getting my newly created Terms page to come up when the terms linked is clicked.  I may have errors in my code.  I still struggle with the syntax of URL rewriting.

Thanks,

Paul

Original URL from Terms Link
http://www.jupiterchristian.org/Home/tabid/53/ctl/Terms/Default.aspx

New URL for newly created Terms page  (Page Title is Terms of Use)
http://www.jupiterchristian.org/TermsofUse/tabid/196/Default.aspx
Sites URL entry  (Input)

<rewriterrule><lookfor>.*Terms-of-Use.aspx</lookfor><sendto>~/Default.aspx?tabid=196</sendto></rewriterrule>

This link works on your site:  http://www.jupiterchristian.org/Terms-of-Use.aspx which tells us that you have the SiteUrls.config updated correctly.

Now all you need is a PageBlaster replacement rule to replace the output of the old link in the page:

<rule><ruleName>Terms Of Use</ruleName><searchFor>hyp Terms(.*?http://.*?)/.*?tabid/\d+/ctl/Terms/Default.aspx</searchFor><replaceWith>hyp Terms$1/Terms-Of-Use.aspx</replaceWith></rule>

Thanks John...worked like a charm. 

Paul

John,

...finally got around to doing this, too, and it works like a charm, save my custom login and registration pages?!? Could the ?returnurl= be screwing this up and require a second rule?

I tried the wildcard, but I don't think I am using the proper syntax.  Am I out in left field on this one?

Hi John,

Can you give an SEO example for just a normal page within the site?  I have been able to follow the tutorial and get my terms and privacy pages done, but don't quite understand what syntax is needed in the pagebalster.config file for the output side.  For example:

I have my sites.url entry for this page set up like this:

http://www.jupiterchristian.org/Athletics/tabid/126/Default.aspx

 
      .*athletics.aspx
      ~/Default.aspx?tabid=126
   

If I type this address into the address bar, the page comes up as expected.  However, when I hover over the actual link it still show the original url.  What entry would I need to add to my pageblaster.config file to change this to www.jupiterchristian.org/athletics.aspx

Armed with this knowledge, I can then make the rest of the pages within my site SEO friendly.

Sorry if you've already answered this somewhere else.

Thanks,

Paul

Hi Paul,

That is described in step #3, changing the output links in the pages.

For your example you would do:

searchFor:  Athletics/tabid/126/Default.aspx

replaceWith:  athletics.aspx

I don't know if you know, but DNN 4.8 has the ability to do all of these automatically now.  You just need to enable Human Friendly Urls.

Thanks John...I've been out of the loop for awhile and was not aware of the new 4.8 feature. 

Paul

Hi Michael,

To switch to HumanFriendly Urls you add an attribute to the FrienlyUrl provider in web.config:

< add name="DNNFriendlyUrl"
                  type="DotNetNuke.Services.Url.FriendlyUrl.DNNFriendlyUrlProvider, DotNetNuke"
                            includePageName="true"
                            regexMatch="[^a-zA-Z0-9 _-]"   urlFormat="HumanFriendly" />

Thank you so much.  That was quick and easy.

Michael