Google Deadly: Kills Small Business
with June 2003 Ranking Rewrite

R. H. Nilsson, Feb 2004

We didn't want to get rich.

We were just a couple of out-of-work programmers trying to make a little extra income. It was an aim with merit. A dream realized legitimately by thousands around the world: put up a small Internet store and generate at least some beer money and enough to pay for the site. All that was needed was a Web domain, a dropshipper (a warehousing company that ships direct to customers from emailed orders), a shopping basket service, and secure credit card processing.

We spent a few days designing a look-and-feel and about a week creating the backend database programming. Nothing fancy, just PHP scripts feeding data from MySQL to the Web page via http query links. In parallel with that effort, we searched the Web and discovered Mal's free shopping cart and a well-recommended card processor. We also created a PayPal account. Finally we had a one-page e-commerce site with a shopping experience like walking into a store. No logins, no games, no email-mining scams, just a store.

The site consisted of one index page, with a few others like "About", "Support" and a "Policies" page, and one that gave a brief description of the kinds of items in each product category -- like a store directory at Leowes or WallMart. PHP functions on the single home page performed all the operatons required for displaying products and linking to the purchasing tools (shopping cart and card processing). One home page script listed product categories which allowed customers to "browse" 20 items at a time in paged list. It was like going down an isle in a large department store where items are grouped by their similarities. Another script displayed weekly specials randomly selected from the database and priced below the normal margins. Yet another script showed detailed pictures and product specifications of an item selected by the customer when s/he was ready to make a choice and actually buy. There was a sophisticated search function built in that allowed narrowing choices using keywords like at Google. Another set of links with pre-set keywords used the search engine for finding common items (like, "show me all the boom boxes" or "digital cameras"). It was essentially a one-page e-commerce application done in PHP. Each of our approximately 6000 product items could be found from this one page. We were proud.

Now, Live and In Color..

The site (we'll just call it "GZ") went live May 1st, 2003 after a third week of testing and debugging on our personal servers. Over the first few days the statistics indicated a growing number of visitors coming to GZ from various search engines, most notably Google.com. This was not surprising, since at that time Google.com was supplying 85% or more of the Web's search results. It felt like the familiar TV commercial depicting an avalanche of customers flooding a company's network services. Well, almost.

Our stats software indicated that Google, along with the other search engines (MSN, Inktomi, etc) were crawling GZ's dynamically generated pages and indexing our roughly 6000 dynamic links, as if each item had its own page. We had several other sites linking to GZ that appeared high in the Google "ranking" for their own categories. Mostly they were "business card" and open information sites. It was obvious from the stats that these were generating referrals and influencing Google's ranking of GZ. The site didn't ever come up #1 in a search, but on several occasions we were on the first page of results, and frequently among the first two or three pages. Not bad for a small "mom & pop" operation.

By the end of May 2003, we were getting an average of 80 visitors a day with about 3 to 5 consummated orders a week. That meager income, although not yet enough to cover our personal design time in terms of wages, at least covered our $90 or so monthly site expenses (hosting, card services, etc.). Our stats showed that 0.5% of visitors actually made purchases, in line with national averages we had seen published, and that was the number we had used to plan our growth for break-even in three months, including paying wages for programming time.

Statistics continued to improve through the first two weeks of June, with no changes to the site design or programming. Projection of growth showed we would break 200 visitors a day by the beginning of July, a 200% per month escalation. Google appeared to be hitting the site at least once a week, with a major crawl (100's of "pages") at the end of the month. The Google ranking tool for Windows Internet Explorer 6 showed a ranking of "6" out of 10 (we thought), which we figured was fair.

Where'd Everybody Go?!

By the middle of the second week of June we had accumulated a total exceeding that for all of May, and were on a steep improvement curve. Then, something horrific happened: site traffic all but died. Mid-June 2003 our daily unique visitors dropped to less than 10 a day, some days even down to 2 or 3, one of which was Inktomi. Now, almost no traffic. Our ranking tool showed a "0".

What could be wrong? We studied the statistics. Although Inktomi, MSN and others still grabbed a lot of bandwidth indexing our product images and some pages consistent with their past activity, Google was no longer crawling our site as aggressively as before. Each Google visit was a single hit of only 45KB of data; about enough to display the front page. This continued until the end (we let the site and domain expire in mid-February 2004, less than a year after going live.)

We had read that it was possible to get onto a Google "blackball" list -- a technique Google uses to thwart would-be "ranking thieves" who use execessive keywording, commented keywords, invisible text, redirection to other sites and such sneaky means to attempt ranking improvement. We had never used any of these techniques. We never had more than about 20 keywords in the keyword meta tag. It didn't make sense to, because folks would be looking for specific item names, features or manufacturers. Searches on verbatim data in our pages resulted in no listings at all for our domain, indicating we had dropped off Google's index altogether.

We emailed Google, asking for help in determining any coding problems. The response said that dynamically generated pages were definitely crawled and that the most important considerations were relevant content and valid URLs in links. We wrote again several times asking if using static pages would help, or if we were on a "keyword spammers" list or something. Google responded with vague suggestions about using their on-line customer forums, and then lapsed into form letters. The forums appeared to be a venue for Google to acquire insight on what techniques web masters were trying to attempt to improve ranking. We made a major effort to enroll in Google's new "Froogle" service by writing scripts to collect our current database into a form they could accept. Google's promotion of that service implied better rankings might result. Over the next few weeks, we found a few errors in HTML page construction that we thought might have caused a misinterpretation of our intent or caused a crawler to hiccup. We corrected them, and... waited.

Over the next few months, after studying competitor's sites, we tried stripping out the "DTD" statement and keywords meta tags altogether, thinking we could jar loose a ranking of "0". No luck. We stripped images from our browser lists, thinking that the time delay for processing images might be causing the crawler to move on to faster sites. Nothing we did made any difference in Google's treatment of GZ, although Inktomi became overly agressive and had to be excluded from our images directory with a "robots.txt" file.

Then we thought, "Well maybe it was just because the country went into out-of-school mode?" -- maybe summer fun activities were robbing e-commerce of attention. A Web search (using Google!) turned up some articles and postings from irate small site owners with similar problems. We weren't the only ones with dropping results statistics. In September 2003 there was a slight growth in visits, but not enough to produce a significant increase in purchases. In October 2003 we tried generating some 40 static pages for the major categories of products to see if Google would crawl them. They were not crawled. Google continued to visit only the front page, not any of the links on it. Along about this time we saw a threefold increase in visits, however we attributed this to the opening Chrismas holiday buying season. Sure enough, in late December, visits dropped dramatically to previous low levels.

We looked at a wide range of sites of different sizes, and came to the conclusion that the larger distributors, even ones that appeared to be using the same dropshipper as we, were getting preferential ranking. It looked like only those sites with large budgets for listing with expensive commercial price-watch services were getting the best rankings.

From Cornucopia to Black Hole

We've come to the conclusion that in June of 2003, Google made a major alteration to its ranking tools (crawlers, spiders, linking indexers, etc.) that has resulted in the demise of at least one mom-n-pop cottage e-commerce site by showing bias for larger commercial organizations including those that buy the pay-by-click links accompanying search results. We're not the only ones -- it appears from posts in the discussion boards of several search optimization services that similar experiences abound among small business storefronts on the Web. Could this have any bearing on the news that Yahoo! is dropping Google as its search results supplier in favor of a home-grown service of its own? Maybe this will start a trend away from Google's 80%+ market share. Perhaps Google has gone the way of other firms that peaked with corporate critical mass and got tangled in their underwear in an effort to increase investor return.

Copyright 2004 Richard Nilsson. Verbatim copying and redistribution of this entire article are permitted without royalty in any medium provided this notice is preserved.