Robots

Description

Robots defines whether to use robots.txt and <META NAME="robots"> exclusions.

Setting Robots to no can be useful when running mnoGoSearch for site validation purposes, as well as when indexing your own Web site.

The default value is yes, for polite crawling purposes.

Scope

Robots can be used multiple times and affects on all following Server and Realm commands until the end of the configuration file, or until the next Robots command.

Examples

Robots no
      

See also

CrawlDelay, Disallow, Index, IndexIf, NoIndexIf, UseSitemap.