Recently, we’ve redesigned our corporate website. The current site is not yet final as we are still lacking some pages for our new services. I am personally working on the content structure and decided to outsource content production to my favorite copywriter. (You can find our old website here)

The main objective for this revamp is to increase our search engine ranking and our conversion rate. So far, so good. Although we are already ranking on the top 5 spots for keywords like web development philippines and seo philippines on Google, Yahoo and MSN, I noticed a big drop on web design philippines keyword (from rank #10 to spot #26, we’re back to #10 spot today but still hoping for #1).

While fixing our drop ranking problem, I noticed that a lot of our client demo websites are being crawled by Google and appears on search results. To fix this problem, I had to create a robots.txt (which I thought was unimportant).

Robots.txt are used by search engine crawlers to check on which directories and files should and shouldn’t be indexed. In-depth explanation on how it works can be found on the robots.txt website.

You can create the file using any text editor and just upload it to the root directory of your website. The basic stuffs that you should know (the ones I used to block some directories from being crawled) are User-agent and Disallow.

sample content of a robots.txt file:
User-agent: *
Disallow: /d*
Disallow: /name_of_directory/
Disallow: /name_of_filename.file_ext/

on the User-agent line, you may specify the crawler name (e.g. googlebot) or just use * for all crawlers. On the disallow line, you have to specify the names of the directory of files that you don’t want crawlers to index. Disallowing d* (Disallow: /d*) tells the crawler not to index all directories that starts with the character d.

You may find additional information on how you can improve your robots.txt on the Google Webmaster page.

I am currently blogging at Cordillera Cofee in Riverbanks. They provide free and fast Wi-Fi access and good food and coffee!