robots.txt is a plain text file placed in the web root directory to give instructions to internet robots on what is allowed to index and what is not allowed to index on the web site. It is a very simple protocol formed with the first directive for User-agent and the next directives is files or directories to exclude from robot indexing.
Examples
To exclude all robot from all files:
User-agent: * Disallow: /
To exclude specific directories from all robots:
User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /js/
To exclude a specific file from all robots:
User-agent: * Disallow: /file.html
To exclude a specific robot:
User-agent: Nasty Robot Disallow: /
