1. Home
  2. Knowledge Base
  3. Web Hosting
  4. What is the robots.txt file?

What is the robots.txt file?

The “robots.txt” file is just a way to telling search engine robots what to and what NOT to index. You are perfectly safe not having one on your site.

The real catch-22 is that legitimate search engines like Google all know how to use the robots.txt file and follow the rules correctly.

There are other robots out there that are designed to collect e-mail addresses and other personal information from web pages. These search engines usually don’t even pay attention to the robots.txt file. Therefore, having one in place doesn’t really help the security of your site.

The only sure way to keep a search engine from indexing parts of your web site is to either not link it in to the rest of the site and/or password protect it.

Was this article helpful?

Related Articles

Need Support?

Can't find the answer you're looking for?
Contact E Street Support