This page will give some basic ideas and concepts for creating a spider friendly site. This is by NO means definitive. There are many great sites out there that discuss the topic of Search Engine Optimization (SEO) in GREAT depth – this isn’t one of them – just a good general overview.
Building a Search Engine friendly site
Choose a good domain name. Right off the bat, that is your best headstart on achieving good rankings. Having a domain name that includes the words that your site will be focusing on is important. It isn’t always possible though. One may need to balance the helpfulness of the words in the title vs. the “fat finger” effect.
When the domain name gets TOO long, it is going to be a problem for some users to type THAT many characters without screwing something up. Example: yahoo.com. Most people can type that without turning it into yqhoo or yshoo or uahii or some other “fat finger” typo problem. Now compare that to arbitrary domain name “myreallylongdomainnamethathaslotsofinfo.com”. Nobody will EVER get that typed in with any consistency without fouling it up. Balance the words in the title with the length.
Know the product that your site is promoting. Product expertise is not necessary but a good healthy understanding of the product; knowing how, when and why customers will want to visit this site; and also knowing what other sites of a similar nature are offering will go a long way towards helping a webmaster build a competitive site.
Learn what terms users seeking this product will be searching with through the search engines. Optimization really focuses on making sure that the terms that will be used in searches are present on your pages. Inserting frequently searched terms will really help your site rankings.
Try to work in these terms in a natural sounding way, not just arbitrarily plopping them in there and making the text sound forced. Ideally the site owner will provide the content and the webmaster can go through the content and retrofit some of these terms in strategic positions.
Add good content. Use text that explains your company/product fully. The more text, the more there is to spider and possibly get found via the search engines. Use images to assist in explaining or describing your product, but don’t alam nothing but images on a page and expect to get any sort of ranking. While images do get spidered, there is only so much information that a text based spider is going to get out of a bunch of images.
There has been a proliferation of Flash sites, but they still are not spidered well IMHO for search engines. At best, a few of the words will be spidered. If you want good ranking, I still suggest you rely on good ol’ text to get the job done. If you’re a well-known rock band and everybody knows your domain name, don’t worry about it, but if like most companies, you need to get your name out there and bring customers that didn’t know you before into your site, use text.
Try to get the most important terms to the top of your page. Search engines seem to favor terms that occur towards the beginning of the body text. That is to say, if your big search term is going to be “custom widgets”, make sure that “custom widgets” appears as soon as possible after the body tag, preferably in an <H1> or similar tag.
Use correct HTML syntax. This is a given. This will ensure that proper alt tags are used, tags are nested properly, etc. When a site is spidered, it is important that it not get bogged down with errors. Just because something looks right on the page doesn’t mean that it IS correct. Always check your HTML to ensure, if not PERFECT compliance, at least very good compliance. See the HTML tips elsewhere on this site.
Include appropriate content and markup for visitors with respect to the WCAG guidelines. This will also help not only your visitors in general, but ensure compliance for handicapped visitors. I try to meet as many of these guidelines as possible when I build sites. The extra content needed for these guidelines gives more to the search engines to spider. Some sites can’t meet these guidelines as easily as others, but at least keep the guidelines in mind.
Add appropriate content to your header and META tags. META tags are specifically there to help explain the content of your site in a rather abbreviated form and this content is useful to search engines. Not all spiders use all this information, but provide it anyway. There are several optimization tools available for researching the kinds of terms, phrases and words that should be included in the your tags. The content of the page should be well reflected in these tags. In a nutshell, the tags should briefly reflect the content that appears on each page. Do NOT use the same tags for each page as each page should be different. I normally use some things common to all of a site’s pages in these tags, but then add specific terms for each page. Include a good title for each page that includes some of those terms mentioned above. Go through the content you have already entered and pick out the important words. Make sure they are represented in your META tags.
ALSO contemplate adding in common misspellings of words on your page. If your content includes commonly misspelled words (Mississippi vs. Missisipi), use them in your meta tags. That way if someone searches and either fat-fingers or misspells the word, your site will STILL be fair game. CAVEAT: Put these links at the END of your META tags. Remember that the META content shows up in descriptions of the page in search engines. Since the content length is limited, the misspelled words won’t show unless someone does actually misspell the word. It is a turn off to find a bunch of misspellings on a page, but with some planning, you can get the results without looking like an illiterate.
Create a robots.txt file for your site. A robots.txt file is read by MOST search engines to outline what areas of your site are to be spidered. Some ill-behaved spiders ignore this, but in general if you don’t want a site area spidered include it in this file. This file should be placed in the root of your site. Also make sure to add the line <meta name=”robots” content=”index,follow”> to your pages.
Check and verify your site content, tags, METAs, etc. This should be a final walk-through of your site to ensure that the site is ready for search engine submission. Bring up each page. Check your title, your meta tag content by viewing the HTML source, verify that you have appropriate tags on all images and all attributes are properly filled in for images, multimedia content or linked documents. Check your links within the site as well as any outbound links.
Get your site submitted to as many free search engines as possible. Submit to any search engine that has content of your nature. Some engines are specifically geared towards certain content. If your content isn’t among those of the search engine, you’re wasting your time. Definitely submit your content to Google, submit to Microsoft Bing and Yahoo, DMOZ (DMOZ has a painful submission process – be prepared) , and any others that are applicable. Whereas there were previously many different search engines not too long ago, most search sites actually use the above engines to power their own services or at least complement them.
If you have a budget, consider submitting to some of the paid search engines. I realize everybody has to make money. It is painful however to have to spend money just to get listed on some of these engines. Paid inclusion is one thing, but just to get listed? Yeesh. Anyway, if you do have some money for this kind of thing, get into Yahoo at least.
These are just the basic steps to getting good content in place and making your site search engine friendly. There are certainly other steps to optimizing your site, but these are some good general things to remember.