Many webmasters noticed that MSN bot rarely or sometimes does not crawl webpages that is more that 3 directory levels deep, in addition to the fact that MSN does not like to crawl webpages that looks like dynamically created. Well, the latter problem can be solve by using “mod_rewrite” to change the URL to some form that looks like static HTML pages. There is a lot of literature available in the internet to make your web pages search engine friendly using .htaccess and mod_rewrite.

Going back to our main problem on how to solve the problem to make MSN bot crawl all your webpages. I will suggest two strategies,

1. Use a sitemap that is being link from the index page. Using a sitemap, the deepest directory has been made 2 levels away from the index page. Be carefull not to exceed 100 links per sitemap because Google don`t like that.

2. Use a URL that has this form,
mydomain.com/pagex.htm
where x is a unique number for each page,
instead of this form;
mydomain.com/country/province/town/name.htm
This method is also a good safeguard if the first strategy does not work because it might be possible that MSN bot count the level of the directory in the URL. Using my suggested URL form, MSN bot will always thought that it is crawling in the top level directory. MSN bot will just wander around your website and keep on crawling every link that it sees.

Case Study
Try searching the cached web pages of filipinolinks.com, yhanpolo.com and alleba.com at Google and MSN using “site:URL” query syntax and compare the results of the two search engines. You will notice that only a several percentage of the total webpages are cached at MSN than at Google. This is because the abovementioned websites use the URL form that is unfriendly to MSN.
Now, try searching the cached webpages of pinoysites.org ,which has around 1400 webpages, at Google and MSN. You will notice that almost all of the webpages are cached, even the deepest page in the website because it uses the URL pattern that I suggested. Futhermore, MSN has more cached webpages than Google for this particular website