Robots.txt Standard Search Engine Guidelines

• Robots.txt Standard

Robots.txt is a standard that controls how search robots access websites. Mistakes in robots.txt files can prevent a site being indexed by search engines.

Rules

• Page has noindex meta tag
• Page has nofollow meta tag