Casing the Competition

A COMPETITOR is eating your lunch. You know it. They know it. What you don’t know is how they did it.

Welcome to the murky world of search engine optimization (SEO). Before you throw your hands up in frustration, I have good news for you: You can learn how to reverse-engineer your archrivals’ tactics and join them at the top of the search results heap.

But first you need to understand who your competitors in the search results really are. More often than not it’s the mom-and-pop sites, not the big players, that give you a run for your money. For example, PC Connection’s fiercest search engine competitor is not PC Mall (even though a search in Google for “pcconnection” produces PCMall.com as the number-one search result!). It’s actually Websites such as Findsavings.com and Smartqpon.com that signed up for PC Connection’s affiliate program that are occupying the top positions for “pc connection” in Google — far outranking PC Connection for its own name. Affiliates tend to be the most adept at search engine optimization and the most lax in abiding by the search engines’ terms and conditions.

WHAT NOT TO DO Affiliates that cheat tend to come and go out of the top search results. Only sites that implement ethical tactics are likely to maintain their positions over time. (By the way, you can help expedite the cheaters’ fall from grace by reporting them to Google at www.google.com/contact/spamreport.html or via email at [email protected].)

How do you know if a top-ranking site is playing by the rules? Look for dubious links to the site using a “link popularity” checker such as www.netconcepts.com/linkcheck. Since the number of links is one factor search engines use to determine search position, less-ethical Websites will link to a multitude of unrelated sites. For example, the current top result in Google for “ionic breeze” (www.smokeaway.org/asseenontv) has inbound links from sites selling debt consolidation, supplements, Levitra, and “business opportunities.”

This sort of sleuthing can reveal some surprises. For instance, I unearthed two devious link schemes:

• GiftCertificates.com’s nemesis is FindGiftCards.com, which commands the top two spots in Google for the all-important search term “gift certificates,” thus relegating GiftCertificates.com to the third position. How did FindGiftCards.com do it? It operates a sister site, 123counters.com, with a free hit counter that propagates “link spam” across thousands of sites that all link back to FindGiftCards.com.

• CraigPadoa.com was a thorn in the side of SharperImage.com, outranking the latter for its most popular product, the Ionic Breeze, by frameset trickery and guestbook spamming (in other words, defacing vulnerable Websites with fake guestbook entries that contained spammy links back to its own site). As soon as The Sharper Image realized what was happening, it jumped on the wayward affiliate. It also restricted such practices in its affiliate agreement and stepped up its monitoring for these spam practices.

SEEKING THE BEST Obviously you don’t want to pull similar tricks. Rather, a good competitor to emulate (or “embrace and extend” as Bill Gates would put it) is one that consistently dominates the upper half of the first page of search results in Google and Yahoo! for a range of important keywords that are popular and relevant to your target audience. But your “mentor” competitors shouldn’t just be good performers; they should also demonstrate that they know what they’re doing when it comes to SEO. More specifically, those rankings shouldn’t be dumb luck. To assess competitors’ competence at SEO, you need to answer the following questions:

• Are their Websites fully indexed by Google and Yahoo!? In other words, are all their Web pages, including product pages, making it into the search engines’ databases? You can use the free tool at www.netconcepts.com/urlcheck to find out. A competitor with only a small percentage of its site in Google probably has a site that is unfriendly to search spiders. A site that isn’t well indexed by Yahoo! might benefit from using the Yahoo/Inktomi Paid Inclusion program (http://www.inktomi.com/products/web_search/sms.html). Paid inclusion has the benefit of 48-hour reindexing, which will allow you to make changes quickly (and thus test hypotheses); but it can get expensive for large sites because the pricing is per page. Google doesn’t offer paid inclusion. As far as I’m aware, the closest thing to Google paid inclusion is Performics’ Virtual Feed.

• Do product and category pages have keyword-rich page titles (title tags) unique to the each page? You can easily review an entire site’s page titles within Google or Yahoo! by searching for “site: www.abccompetitor.com.” (Incidentally, this type of search might also yield confidential information. A lot of Webmasters don’t realize that Google has discovered and indexed commercially sensitive content buried deep in their site. For example, a Google search for “confidential business plan filetype:doc” will yield a lot of real business plans among the sample templates.)

• Do product and category pages have reasonably high PageRank scores? PageRank is the numerical representation of how important Google considers a site to be and thus, in part, how well it will rank. (Inktomi has a rough equivalent to PageRank called LinkFlux; however, it is not quantifiable by observers outside of Inktomi Corp.) You can check a page’s PageRank score using the Google Toolbar, which is available as a free download at http://toolbar.google.com. Note that the toolbar does not always return the true PageRank. The toolbar only queries Google’s toolbar server if it’s the first page of that site you have visited since you last exited your Web browser. For subsequent pages within the same visit it will give an estimated ranking.

• Is link text across the site, particularly in the navigation, keyword rich? Google associates the underlined hyperlink text with the page that’s being linked to. “Click here” and “Read more” are not particularly helpful to search rankings.

• Are the Websites getting “filtered” out of the search results? You can overdo SEO. Too much keyword repetition or too many suspiciously well optimized text links can yield a penalty for overoptimizing. Check whether a site is being filtered using Scroogle (http://www.google-watch.org/scraper.html). For instance, at the time of writing this article, Bluefly.com was being filtered out of the first page of search results for “gucci.”

• Are they link-spamming the search engines with “doorway pages”? A doorway page is a Web page full of keyword-rich copy that doesn’t deliver any useful information on it other than a link into the site; its sole purpose is to be fed to the search engines. Search engines strongly advise against using this ploy. You can sometimes spot doorway pages by reviewing a site’s complete list of pages (use the URL Checker tool at www.netconcepts.com/urlcheck). Both JCPenney.com and PCConnection.com, for instance, use doorway pages, unfortunately for them. These doorways are buried at the bottom of the search engine results.

UNCOVERING THEIR SECRETS Let’s assume that your investigating has left you with several competitors who are gaining excellent search placement using legitimate, intelligent tactics. Now it’s time to uncover their secrets:

• What keywords are they targeting? You can determine this by looking at the page titles (up in the blue bar at the top of your Web browser, which also appears in the search results listings) across their home page and product category pages, then by looking at their meta-keywords tag (right-click, select “View Source,” then scour the HTML source for the list of keywords that follow the bit of HTML that looks something like “meta name=keywords content=”).

  You could also try analyzing the keyword frequency on the page using the free tool at www.keywordcount.com. Also note keywords purchased when you see your competitor appear in the pay-per-click search ads on the search results of Google, Yahoo!, and the like.

• Who’s linking to their home page? To their top-selling product pages and category pages? A link popularity checker (such as www.netconcepts.com/linkcheck/) can be quite helpful. Amazon.com has been especially successful in achieving deep links into its millions of products, even when discounting the multitude of affiliate links. (Third-party-hosted affiliate networks such as LinkShare do not transfer PageRank score from affiliate links to the merchants’ sites.)

• What technology tricks are they using to get search engine spiders such as Googlebot to cope with the site being dynamic (database-driven)? Nearly all the technology tricks are tied to the e-commerce platforms the competitors are running. You can check to see if they are on the same server software as you by using the “What’s that site running?” tool on the top left corner of www.netcraft.com. Both LLBean.com and Sears.com, for example, use Netscape Enterprise server version 4.1. Look at “cached” (archived) versions of your competitors’ pages by clicking on the “Cached” link next to their search results in Google to see if they’re doing anything too aggressive, such as “cloaking” (where they serve up a different version of the page to search engine spiders than to human visitors).

• What affect will their future SEO initiatives have on their site traffic? Assess the success of their SEO not just by the lift in rankings. Periodically record key SEO metrics over time — the number of pages indexed, the PageRank score, the number of links — and watch the resulting effect on their site traffic. You don’t need access to competitors’ server logs to get an idea of how much traffic they are getting. Simply go to www.alexa.com, enter the competitor’s domain into the search box, and then on the subsequent page, click on “See Traffic Details” to view a chart of the site’s traffic history based on Alexa toolbar users as a sample of total users. You can even superimpose your own traffic history onto your competitor’s chart.

• How does the current state of their sites’ SEO compare with those of years past? You can reach back into history and access previous versions of your competitors’ home pages and view the HTML source to see which optimization tactics they were employing back then. The Wayback Machine (http://www.archive.org) provides an amazingly extensive archive of Web pages. A search of the Wayback Machine for Cabelas.com yielded nearly 200 versions of the site dating as far back as 1996 (http://web.archive.org/web/*/www.cabelas.com).

Stephan Spencer is founder/president of Netconcepts, a Madison, WI-based e-marketing agency, and coauthor of the Catalog Age special report “State of Search Engine Marketing for Retailers 1.0.”

Your competitors can be your best teachers