Robots are programs that scan the web automatically, for different reasons: search engines such as Google™, for example, use them to index website contents; spammers, on the other hand, use them to obtain e-mail addresses without authorization.
A website owner uses the robots.txt file to tell robot programs what they are supposed to do with his website: robots (at least the trustworthy ones) check for the presence of a robots.txt file and follow the instructions in it, before accessing a website.
So, the robots.txt file is simply a text file with a list of instructions that specify:
|1.||the kind of robot that the rules apply to|
|2.||the URLs of the pages to be blocked.|
WebSite X5 provides a default robot.txt file which contains instructions so that no robots should consider the contents of certain folders, such as the admin and res folders:
You can change these basic instructions, according to your specific requirements.
To edit and include the robots.txt file, you must:
|▪||Open the Statistics, SEO and Code | SEO window and select the Include the robots.txt file option.|
|▪||Use the field underneath to manually edit the instruction to include in the robots.txt file.|
For more information on robots, and on how to create a robots.txt file, see the official website http://www.robotstxt.org or consult Google™ Webmaster Central (Block or remove pages using a robots.txt file)