Is Google Missing Your Web Site?
- Find Out Where You Stand
- Eliminate Spider Traps
- Create Spider Paths
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
If it seems like everyone is using search engines these days, you're paying attention. And those searchers are exactly the kind of the people you want to visit your Web site—one-third of them are shopping for something and over 40% research purchases. But what if your site isn't found in Google? How do you get in on the action?
Many problems can thwart your site's visibility in search engines, but one of the main ones is missing pages from the search index—the index is the database of pages that search engines consult when looking for matches for a searcher's words. If your Web site's pages are missing from Google's search index, they can never be found.
Now, it is relatively rare for a Web site to be completely missing from a search index, although it can happen if a site is very new or if it has been banned from the search engine for unethical practices. But it is extremely common for sites to have only some of their pages indexed—sometimes a very small percentage—which means that many searches that should find pages on your site won't. They'll find pages from your competitors instead.
So, there you sit. You know search is important. You know that your pages must be in the search index to be found. And you suspect that not all of your Web site's pages are in the search index. But what do you do about it?
Find Out Where You Stand
Rather than speculating, get the facts. Search engines will happily tell you how many pages are indexed from your site, if you know how to ask. Google, MSN Search, and Yahoo! Search all provide a "site:" operator that can show you the pages indexed (such as "site:ford.com" ). Or you can use the free Saturation Reporting Tool from MarketLeap to tell you how many of your pages are included in each index.
But finding out how many pages you have indexed is only half the story. You also need to know how many pages you have on your site. Now, those of you running small Web sites may be snickering at this point—you know how many pages you have, for Pete's sake. Many medium-to-large Web sites, however, can't pin down exactly how many pages they have. They need to use some method of estimating their total number of pages, ranging from asking their Webmaster to taking a guess to counting the number of documents in their content management systems. You can unleash special programs on your site, such as the free Xenu or the $98 OptiSpider, which are designed to find pages that you might have overlooked. As a last resort, you can use the highest number of pages you found in any search engine using the MarketLeap tool.
Regardless of how you estimate your site's total pages, your next step is to calculate your inclusion ratio, the percentage of your total pages that are indexed. So, if you estimate your site to have 500 pages and Google has indexed 200 of them, your Google inclusion ratio is 200 divided by 500, or 40%. All of which begs the question, "What is a good inclusion ratio?" You can get by with about 50% of your pages indexed, but it is possible to get nearly 100% indexed if you do the right things. Some sites have just 5% of the pages indexed, so traffic from search engines can be radically improved with a few of the right moves. Let's look at what you can do.