Invite Search Engine Spiders Into Your Dynamic Web Site
- Date: 2008-07-03 - Word Count: 1256
Share This!
But did the spider actually visit? Maybe not. Dynamic page content is often invisible to most search engine spiders, so it never gets indexed. Increase the traffic to your dynamic site by making your valuable content visible to search engine spiders.
Dynamic Pages Are Easy To Maintain
The content of static pages doesn't change unless you actually code the changes into your HTML file: open the file, edit the content, save the file, and upload it to the server. All search engine spiders can index static Web pages.
A dynamic Web page is a template that displays specific information in response to queries. Most of the page content comes from the database connected to the Web site. Visitors love them since they get quick access to the information they want. These sites are easy for webmasters to update: as product offerings or prices change, just edit your database instead of hundreds of individual Web pages.
Search engine spiders have a much tougher time with dynamic sites. Some get stuck because they can't supply the information the site needs to generate the page. Other spiders deliberately stay away from dynamic pages to avoid getting trapped in the site.
What Was The Question Again?
Visitors find information in a dynamic site by using a search query. That query can either be typed into a search form by the visitor or already be coded into a link on the home page - making the link a pre-defined search of the site's catalog. In that later case, the portion of the link containing the search parameters is called a 'query string.'
But a search engine spider doesn't know to use your search function - or what questions to ask. Dynamic scripts often need certain information before they can return the page content: cookie data, session id, or a query string are common requirements. Spiders usually stop indexing a dynamic site because they can't answer the question.
If the spider does accidentally wander deeper into your site, it could inadvertently get caught in a "spider trap": a badly written CGI script that requests information the spider can't supply. Then the spider and your server navigate a never-ending loop where a request for a page is met with a request for information.
Getting a spider trapped inside your server is not just bad for the spider. The repeated requests for pages can crash the server. If you share server space with other Web sites and have a problem with site downtime, ask your Web host to check for CGI script problems on other sites.
It's All In The Name
A page's actual URL address often poses a problem too because most dynamic page URL's contain query strings. Here's an example of the URL for a book search result page on Barnes and Noble's Web site:
http://shop.barnesandnoble.com/booksearch/isbnInquiry.asp?
userid=2IMXLT5XN1&mscssid=QEUFGRFF5X2G9H2UCMJQLAKJ8JV83FMD&isbn=0452269350
Look closely at the URL. See the question mark after /isbnInquiry.asp? Most search engine spiders get to the "?" in the query string and stop indexing because of the probability of getting caught in a spider trap.
Attract Spiders To Your Web
So, you've got all this invisible content - what do you do? Search engines know about the problem, but most have shown very little interest in addressing it. Infoseek and HotBot are the exception. Their search engine spiders can index dynamic page content, but don't do it automatically. You have to invite them in.
HotBot recommends that you submit your dynamic page with all the arguments added onto the URL (www.website.com/products/search/product_query.asp?prod_id=22929). You can also submit a static page that contains links to the dynamic URLs you need indexed.
Infoseek's spider, called Slurp, will index dynamic pages that you submit, but won't crawl through your dynamic Web site by default.
You do have options to get indexed by the other search engines, but no matter which you select, you'll have to spend some time and effort to make sure your dynamic content gets indexed.
Add Dynamic Links To Static Pages
Include links to important dynamic content on your static pages. The simplest way is a straightforward table of contents page that links to your most important dynamic pages. It gives spiders a way to index content without having to answer any questions. If you have a small site with few products, this is a quick way to get more of your content indexed.
However, the table of contents won't help you with search engine spiders that stop at query strings. Increase your chances by including good, descriptive links to your major product categories on a static products page. Search engines that stop at query strings will still index the content of the products page - including your link titles. Other search engines that can follow dynamic links can visit the actual dynamic page content without a query.
Remove Query Strings From Dynamic URL's
Amazon.com uses this method to get its product selections indexed by search engines. For instance, a search on Google for Rachael Carson's book, Silent Spring, returns a result that takes you directly to the appropriate dynamic page at Amazon: http://www.amazon.com/exec/obidos/ISBN%3D0395683297/103-0475212-8205437. Because the URL doesn't contain any query strings, all search engines can index Amazon's product line.
This method works, but it's also the most technically demanding solution. If you decide to use this method, you can select from several different options, depending on the type of Web server you use and the software you're using to integrate your database with your Web site:
Cold Fusion: Reconfigure your Cold Fusion setup to replace the "?" in a query string with a '/' and pass the value to the URL. The browser interprets that as a static URL page.
Instead of http://www.mystore.com/products.cfm?prod_id=22343, you get a string like this: http://www.mystore.com/products.cfm/22343.
CGI Scripts: Path_Info (or Script_Name) is a variable in a dynamic application that contains the complete URL address (including the query string information). Write a script that strips out all the information before the query string and set the balance of the information equal to a variable. You can then use that variable in your URL address.
Apache: has a special rewrite module that allows you to translate URL's containing query strings into URL addresses that search engine spiders can follow. The module, mod_rewrite, isn't automatically installed with Apache software. Check with your Web host or administrator and see if it's available on your server.
Visit the Apache Web site for more information on the mod_rewrite module.
Active Server Pages: Most search engines will index .asp pages if the "?" is removed from the URL. XQASP offers a product that will automatically remove the query strings from your .asp pages and replace them with "/" marks.
A note of caution: these four methods make your dynamic page appear to have its own sub-directory, so the browser will look for images and links there. You can completely avoid broken links and pages by using all absolute URL addresses on your page, but that will make maintenance more difficult later. Alternatively, use URL addresses that are relative to the root directory of your site, not the document. Use /homepage.htm instead of ../homepage.htm and you'll be fine.
Remember The Rules
Don't get so caught up in modifying your page design or URL addresses that you forget the basic rules for search engine optimization. Your pages need to have good content, META tags, a high link popularity score, appropriate keywords, and more before you can climb to the top of the search engine ranking.
NetMechanic's Search Engine Power Pack helps
you get all your pages ready to submit to search engines. It contains the tools you need to get and keep a top ranking.
For more detail visit us at http://webdesignerhouston.com
(Our Services includes: Houston website design, Houston search engine optimization, Houston seo services, web design services Texas, professional website designer )
Dynamic Pages Are Easy To Maintain
The content of static pages doesn't change unless you actually code the changes into your HTML file: open the file, edit the content, save the file, and upload it to the server. All search engine spiders can index static Web pages.
A dynamic Web page is a template that displays specific information in response to queries. Most of the page content comes from the database connected to the Web site. Visitors love them since they get quick access to the information they want. These sites are easy for webmasters to update: as product offerings or prices change, just edit your database instead of hundreds of individual Web pages.
Search engine spiders have a much tougher time with dynamic sites. Some get stuck because they can't supply the information the site needs to generate the page. Other spiders deliberately stay away from dynamic pages to avoid getting trapped in the site.
What Was The Question Again?
Visitors find information in a dynamic site by using a search query. That query can either be typed into a search form by the visitor or already be coded into a link on the home page - making the link a pre-defined search of the site's catalog. In that later case, the portion of the link containing the search parameters is called a 'query string.'
But a search engine spider doesn't know to use your search function - or what questions to ask. Dynamic scripts often need certain information before they can return the page content: cookie data, session id, or a query string are common requirements. Spiders usually stop indexing a dynamic site because they can't answer the question.
If the spider does accidentally wander deeper into your site, it could inadvertently get caught in a "spider trap": a badly written CGI script that requests information the spider can't supply. Then the spider and your server navigate a never-ending loop where a request for a page is met with a request for information.
Getting a spider trapped inside your server is not just bad for the spider. The repeated requests for pages can crash the server. If you share server space with other Web sites and have a problem with site downtime, ask your Web host to check for CGI script problems on other sites.
It's All In The Name
A page's actual URL address often poses a problem too because most dynamic page URL's contain query strings. Here's an example of the URL for a book search result page on Barnes and Noble's Web site:
http://shop.barnesandnoble.com/booksearch/isbnInquiry.asp?
userid=2IMXLT5XN1&mscssid=QEUFGRFF5X2G9H2UCMJQLAKJ8JV83FMD&isbn=0452269350
Look closely at the URL. See the question mark after /isbnInquiry.asp? Most search engine spiders get to the "?" in the query string and stop indexing because of the probability of getting caught in a spider trap.
Attract Spiders To Your Web
So, you've got all this invisible content - what do you do? Search engines know about the problem, but most have shown very little interest in addressing it. Infoseek and HotBot are the exception. Their search engine spiders can index dynamic page content, but don't do it automatically. You have to invite them in.
HotBot recommends that you submit your dynamic page with all the arguments added onto the URL (www.website.com/products/search/product_query.asp?prod_id=22929). You can also submit a static page that contains links to the dynamic URLs you need indexed.
Infoseek's spider, called Slurp, will index dynamic pages that you submit, but won't crawl through your dynamic Web site by default.
You do have options to get indexed by the other search engines, but no matter which you select, you'll have to spend some time and effort to make sure your dynamic content gets indexed.
Add Dynamic Links To Static Pages
Include links to important dynamic content on your static pages. The simplest way is a straightforward table of contents page that links to your most important dynamic pages. It gives spiders a way to index content without having to answer any questions. If you have a small site with few products, this is a quick way to get more of your content indexed.
However, the table of contents won't help you with search engine spiders that stop at query strings. Increase your chances by including good, descriptive links to your major product categories on a static products page. Search engines that stop at query strings will still index the content of the products page - including your link titles. Other search engines that can follow dynamic links can visit the actual dynamic page content without a query.
Remove Query Strings From Dynamic URL's
Amazon.com uses this method to get its product selections indexed by search engines. For instance, a search on Google for Rachael Carson's book, Silent Spring, returns a result that takes you directly to the appropriate dynamic page at Amazon: http://www.amazon.com/exec/obidos/ISBN%3D0395683297/103-0475212-8205437. Because the URL doesn't contain any query strings, all search engines can index Amazon's product line.
This method works, but it's also the most technically demanding solution. If you decide to use this method, you can select from several different options, depending on the type of Web server you use and the software you're using to integrate your database with your Web site:
Cold Fusion: Reconfigure your Cold Fusion setup to replace the "?" in a query string with a '/' and pass the value to the URL. The browser interprets that as a static URL page.
Instead of http://www.mystore.com/products.cfm?prod_id=22343, you get a string like this: http://www.mystore.com/products.cfm/22343.
CGI Scripts: Path_Info (or Script_Name) is a variable in a dynamic application that contains the complete URL address (including the query string information). Write a script that strips out all the information before the query string and set the balance of the information equal to a variable. You can then use that variable in your URL address.
Apache: has a special rewrite module that allows you to translate URL's containing query strings into URL addresses that search engine spiders can follow. The module, mod_rewrite, isn't automatically installed with Apache software. Check with your Web host or administrator and see if it's available on your server.
Visit the Apache Web site for more information on the mod_rewrite module.
Active Server Pages: Most search engines will index .asp pages if the "?" is removed from the URL. XQASP offers a product that will automatically remove the query strings from your .asp pages and replace them with "/" marks.
A note of caution: these four methods make your dynamic page appear to have its own sub-directory, so the browser will look for images and links there. You can completely avoid broken links and pages by using all absolute URL addresses on your page, but that will make maintenance more difficult later. Alternatively, use URL addresses that are relative to the root directory of your site, not the document. Use /homepage.htm instead of ../homepage.htm and you'll be fine.
Remember The Rules
Don't get so caught up in modifying your page design or URL addresses that you forget the basic rules for search engine optimization. Your pages need to have good content, META tags, a high link popularity score, appropriate keywords, and more before you can climb to the top of the search engine ranking.
NetMechanic's Search Engine Power Pack helps
you get all your pages ready to submit to search engines. It contains the tools you need to get and keep a top ranking.
For more detail visit us at http://webdesignerhouston.com
(Our Services includes: Houston website design, Houston search engine optimization, Houston seo services, web design services Texas, professional website designer )
Related Tags: seo services, web design services, houston website design, houston seo, houston search engine optimization, professional designs, search engine optimi
Web designer Houston.com provides high quality and professional website design and search engine optimization services to Houston, Austin, San Antonia, Dallas, TX and nationwide Your Article Search Directory : Find in Articles
Recent articles in this category:
- Improper Way Of Marketing Reflects Poorly On A Company.
New business, product or service everything requires visibility, awareness in order to come into the - Replacing Paper Prints With Online Versions
Nowadays saving out on resources and being additionally informative are both aspects that are in. In - Stop Smoking Effectively
If I told you of a way that you could stop smoking harmful tobacco would you believe it? Most people - What Is Runtime Error 182? And How To Fix It
Are you finding an effective way to fix runtime error 182? Do you think fixing runtime error 182 is - Do You Know How To Fix Runtime Error 87 In Minutes?
Are you finding an effective way to fix runtime error 87? Do you think fixing runtime error 87 is to - Knowledge About Avi, Avi Player, Avi Converter On Mac
Knowledge about AVI, AVI player, AVI Converter on MacWhat is an AVI?AVI, an acronym for Audio Video - Buy Your Highly Successful Email Survey Software Today
Email Survey Software- Boost Your Business and Increase ProfitsAn email survey software could be one - Xrm - The Anything Relationship Management Solution
I recently attended the Microsoft Dynamics West Region FY11 Sales Planning Retreat. This year's meet - What Are The Benefits Of Working With Electronic Medical Records
Recording medical information is a vital part of health care services. These records are necessary f - Basic Factor To Make Website Business Oriented
Internet is home for millions of websites. The online business is becoming more and more competitive
Most viewed articles in this category:
- Parental Control Software
Parental control software is software that can help parents protect their children when they are onl - Digital Asset Management Software
Managing and organizing your organization's documents is a critical component to your business's suc - AdobeRGB vs. sRGB
Understanding color spaces I'll try to explain it very simplified, but understandable for everyone - Confessions of a Prankster
I wanted to get a jump on April Fool's Day, partially because of the long, cold winter blues, and pa - Malicious Thoughts About The Spyware Ills Of My PC
Who would think I was capable of such revengeful thoughts about the parties responsible for inflicti - Recover File and Recover Deleted File Tools
Data recovery software is a very effective way of retrieving data from a worn or damaged hard disk d - Life without Windows
Ubuntu, a user-friendly version of Linux, has been running so nicely on my home PC that I decided to - What Benefit Does an Online Software Download Site Offer You?
Are you having a problem that where you find a good softeware when you consider to have a try or wan - Maintaining A Website
There was an era when people were talking about how to create a website using html coding or some ea - Benefits Of Proper Time Tracking
Have you ever written down time when you have started and finished your work? Maybe you have had mul