Your Search Engine Ranking and Robots.txt
If ranking well in search engines is a priority for you, you need to know about robots.txt. It’s something so small and so simple that it’s easily overlooked by even experienced web developers and search engine optimization folks.
A little history
When search engines started reading public websites, they soon realized that some people didn’t want their information shared all over the world. Instead, some websites have a need to keep some or all of their content private.
To accommodate this, reputable search engines came up with a simple solution where they basically said, “You just tell us what you want public and what you want private, and we won’t index the private stuff.”
Robots.txt
Which brings us to robots.txt. Search engines decided that if you make a text file (a file with the extension “.txt”) and name it “robots,” they’ll look at it and do what it says.
They went on to give us a specific way how to tell them what to allow and what not to allow. You can see those specifications here.
The most basic robots.txt file will look something like this:
User-agent: * # applies to all robots Disallow: / # disallow indexing of all pages
The instructions above would tell all search engines to not index any pages on the website.
Some website hosts and software have this setting to disallow all search engines as the default. Others allow you to set it in an admin area (like WordPress or Joomla). Still others allow you to do whatever you want with it, which means you have to create it, define it, and upload it yourself.
Here’s the Rub
How does all of this affect you? If you or your web team has accidentally left the disallow default set or defined the instructions incorrectly, search engines are be leaving you out of their results altogether.
This single, tiny mistake will absolutely obliterate all your other search engine optimization investments.
Check for Yours
So, right now, go check to see if you have a robots.txt in your main web directory. Do this by typing the following into your browser’s address bar:
http://www.put-your-domain-name-here.com/robots.txt
If you don’t have one, that’s okay. By default, search engines will include you in their indexes.
If you do have one, make sure that it doesn’t have any instructions to disallow search engines from adding your site to their search results. It may have the word disallow and that’s okay, as long as that word isn’t followed by anything else, especially a slash “/”.
If you’d like someone to take a look and see if yours looks alright, feel free to give us a call.
