Posted by admin in Recently, we’ve redesigned our corporate website. The current site is not yet final as we are still lacking some pages for our new services. I am personally working on the content structure and decided to outsource content production to my favorite copywriter. (You can find our old website here)
The main objective for this revamp is to increase our search engine ranking and our conversion rate. So far, so good. Although we are already ranking on the top 5 spots for keywords like web development philippines and seo philippines on Google, Yahoo and MSN, I noticed a big drop on web design philippines keyword (from rank #10 to spot #26, we’re back to #10 spot today but still hoping for #1).
While fixing our drop ranking problem, I noticed that a lot of our client demo websites are being crawled by Google and appears on search results. To fix this problem, I had to create a robots.txt (which I thought was unimportant).
Robots.txt are used by search engine crawlers to check on which directories and files should and shouldn’t be indexed. In-depth explanation on how it works can be found on the robots.txt website.
You can create the file using any text editor and just upload it to the root directory of your website. The basic stuffs that you should know (the ones I used to block some directories from being crawled) are User-agent and Disallow.
sample content of a robots.txt file:
User-agent: *
Disallow: /d*
Disallow: /name_of_directory/
Disallow: /name_of_filename.file_ext/
on the User-agent line, you may specify the crawler name (e.g. googlebot) or just use * for all crawlers. On the disallow line, you have to specify the names of the directory of files that you don’t want crawlers to index. Disallowing d* (Disallow: /d*) tells the crawler not to index all directories that starts with the character d.
You may find additional information on how you can improve your robots.txt on the Google Webmaster page.
I am currently blogging at Cordillera Cofee in Riverbanks. They provide free and fast Wi-Fi access and good food and coffee!
Related posts:
- Key factors for a successful website To design a successful website, here are some key factors...
- Beginner SEO Tips A friend is starting a new website and asked me...
- 5 articles you should read if you own an online storefront 12 Tips for Designing an excellent Checkout process by...
- Cosmetics and Dermatological Website We’ve recently launched the new website of Derm Clinic. Derm...
- How to track online comments about your website ConvoTrack lets you load comments from Twitter, FriendFeed, Digg, Reddit,...
[...] forms, log-in forms, carousels, etc. This is a great resource if you are planning your next website redesign. my favorite pricing table design is by MovableType. [From: Pricing Tables showcase] [...]